<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
  xmlns:content="http://purl.org/rss/1.0/modules/content/"
  xmlns:dc="http://purl.org/dc/elements/1.1/"
  xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd"
  xmlns:trackback="http://madskills.com/public/xml/rss/module/trackback/">
  <channel>
    <title>Adam C. Conrad</title>
    <link>https://www.adamconrad.dev</link>
    <description>The website and home of engineering leader Adam C. Conrad.</description>
    <pubDate>Tue, 29 Mar 2022 14:01:00 +0000</pubDate>
    <item>
      <title>Behavioral interviews</title>
      <link>https://www.adamconrad.dev/blog/behavioral-interviews/</link>
      <description><![CDATA[Earlier this month, I mentioned the importance of behavioral interviews for engineering managers. This next series of blog posts is going to tackle behavioral interview questions.

]]></description>
      <pubDate>Tue, 29 Mar 2022 14:01:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/behavioral-interviews/</guid>
      <content:encoded><![CDATA[Earlier this month, I [mentioned](/blog/changing-jobs-during-the-great-resignation/) the importance of behavioral interviews for engineering managers. This next series of blog posts is going to tackle behavioral interview questions.

## Getting started

Because behavioral interviews tackle human interactions, and humans are complex, you can expect a lot of variety in this bucket of questions. Before we explain the aspects of each behavioral question, it's wise to start with foundational preparation for all behavioral questions.

### The STAR method

**The most important thing to remember about behavioral interviews is the STAR method.** STAR stands for **S**ituation, **T**ask, **A**ction, **R**esult.

The _situation_ is the background for how your story is set. Who is involved? How are they connected? Why are you involved?

The _task_ is what you were asked (or decided) to do in light of this situation. What did you have to get done? What did others have to get done? How did you influence others to get things done?

The _action_ is the application of that task. What did you do against what was expected of you? What did others do as a result of your actions? What influence did you impart on those around you?

The _result_ is the outcome of your actions. How did you perform your tasks? Did it resolve the situation? Were there any unintended consequences, good or bad, from what you did in this situation?

This format is the gold standard for answering behavioral questions. While you won't need this for every behavioral question, you can't go wrong if you use it as a fallback.

### Short and sweet

**The second most important thing to remember about behavioral interviews is brevity.** Behavioral interviews do not linger on one question. Many questions are asked in a short amount of time, often with lots of follow-up questions. To ensure interviewers get all of the signals they need, you must answer questions succinctly. The best way to practice this is to keep your answers short and without filler.

Applying both lessons, you should aim to **practice behavioral interview questions using the STAR method, capping your answers at under 2 minutes per question.** It is strongly recommended that you use a timer to ensure that you are forcing yourself to keep time in mind when answering questions. Doing this will ensure you keep your answers short with the ideal format.

### Create an autobiography

Your resume summarizes the highlights of your career. As an exercise, **re-tell your resume in an expanded autobiography**. With each project, answer the following:

* Why were they tackled?
* What were their objectives?
* How were you involved? Who did you work with?
* What were the outcomes of those projects?
  * What mistakes and failures did you make?
  * What successes did you enjoy?
* What roadblocks and conflicts did you hit along the way? How did you overcome them?
* What tradeoffs did you have to make? How did you choose between all of the options?
* What did you learn? What would you do differently today?

Cataloging all of these points across all of your major technical projects throughout your career will provide the necessary information to tackle any question you're asked. With this information captured, it will be **less about preparing for questions and more about aligning the correct situations with the right questions**.

## It's all about me

As a last reminder, don't be ashamed to highlight your specific achievements. It is expected that you will showcase teamwork and interpersonal skills. For these questions, **interviewers want to know what _you_ did**. If someone were to ask you, "what was a time you led a team to a huge success?" this would be a _weak_ answer:

> We had to fix a showstopper bug on our account settings billing page. We had to triage the bug and coordinate with the developers to find the root cause the problem and fix it. Our on-call developer quickly resolved the bug and restored access to this crucial billing page. We estimate it saved the company $100k in lost revenue by implementing these reactive practices.

Why is this a poor answer? It follows the STAR method, it's short, and recounts an actual success you experienced in a previous role.

**The problem is it says nothing about what _you specifically did_ in this situation.** Were you the team lead? The engineering manager? What part of the triage process did you participate in? Did you  perform the triage? Did you implement the triage process documentation for the team? Did you hold a post mortem or root cause analysis to learn from this mistake? Did you run the report to quantify the impact?

These questions will swirl in the interviewer's mind. They will have to spend several minutes picking apart your answer to discover your contribution. This will take away from the other questions they wanted to ask without capturing the signal they were looking for.

**Behavioral interviews are where it is okay to talk about yourself and how you succeeded in the past.** Do not be afraid to use the words "I," "me," or "mine." In the end, only _you_ are changing jobs, not your whole team.
]]></content:encoded>
      <dc:date>2022-03-29T14:01:00+00:00</dc:date>
    </item>
    <item>
      <title>Default to gratitude</title>
      <link>https://www.adamconrad.dev/blog/default-to-gratitude/</link>
      <description><![CDATA[Last Friday I was completely drained. I didn’t have the best day at work, I was sleep deprived, and I wasn’t in the best mood.

]]></description>
      <pubDate>Wed, 23 Mar 2022 23:11:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/default-to-gratitude/</guid>
      <content:encoded><![CDATA[Last Friday I was completely drained. I didn't have the best day at work, I was sleep deprived, and I wasn't in the best mood.

So I decided to start messaging people and thanking them. I went on a gratitude spree.

**Gratitude has an infinite ROI**. It is [well established](https://theconversation.com/express-gratitude-not-because-you-will-benefit-from-it-but-others-might-134887) that gratitude improves your mental health and wellbeing. If you're sharing your gratitude for others, your genuine feelings will build trust and rapport.

There is literally no downside to honest, heartfelt gratitude. Express it often and with full authenticity. **If your brain has nothing left to give, give gratitude.**
]]></content:encoded>
      <dc:date>2022-03-23T23:11:00+00:00</dc:date>
    </item>
    <item>
      <title>Changing jobs during the Great Resignation</title>
      <link>https://www.adamconrad.dev/blog/changing-jobs-during-the-great-resignation/</link>
      <description><![CDATA[The last time I contributed to this blog was exactly 7 months ago. There is a reason it has been so long: I changed jobs.

]]></description>
      <pubDate>Fri, 18 Mar 2022 20:53:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/changing-jobs-during-the-great-resignation/</guid>
      <content:encoded><![CDATA[The last time I contributed to this blog was exactly 7 months ago. There is a reason it has been so long: I changed jobs.

## I got stuck

No good can come from putting your old company on blast. Suffice it to say I was not growing anymore and I needed a change.

Once I knew it was time to leave, I needed a plan to get to my next destination.

## Prep before the prep

This was the longest job I had ever held. I was extremely out of practice interviewing as the interviewee. As an engineering manager, you have a very unique perspective. You understand how hiring decisions get made. So I knew what my new hiring managers know in terms of what to look for. Since I knew what to look for in someone like myself, my first task was to **create a list of companies I wanted to work at** and then **figure out the hiring process for each of them**.

### What I looked for

I started by looking at Levels.fyi. On Levels, I wanted to see what similar profiles were getting offered.

I looked at all of their salary data for engineering managers. I filtered on 11-12 years of experience and only selected new hire offers. Finally, I sorted by highest salary. I then applied to the top 10 companies on that list who either hired remotely or hired in my area (Boston).

All in all, including a few companies that reached out to me, my final list looked like this:

* Amazon
* Brex
* Coinbase
* Cruise
* Dropbox
* Facebook
* Google
* Instacart
* LinkedIn
* Netflix
* Physna
* Plaid
* Salesforce
* Stripe
* Twitter
* YUM Brands

### Doubling down on prep

Once I had my list of companies, it was time to start learning about their hiring processes. This is where Blind came in. Lots of people reveal what is asked at their companies. **They also offer to refer you to their company**. If I could do it again, I would have asked for referrals from all of those companies above.

I cannot stress this enough: **referrals are the most important part of getting your foot in the door**. So much so that the only 7 companies who offered an on-site were companies where I had a referral. Even still, I had referrals at both Google and Salesforce, both of which went nowhere after the initial submission phase.

Based on scouring Blind's forums, I knew I had to focus on two general areas: [systems design](/tag/systems-design/) and behavioral questions. If I wanted a job at Google or Facebook, I'd have to focus on [coding](/tag/algorithms) as well.

## My study plan

Now that I had a plan of attack, I knew I needed time. After spending way too much time watching YouTube videos on [how to prepare](https://www.youtube.com/watch?v=7UlslIXHNsw), I realized I needed at least 3 months to prepare. Because of this, **I didn't apply to these companies right away**. Instead, I knew that the actual interview process, even if it was slow, would still be faster than how long it would take for me to prepare for their interviews.

A major issue in getting started was **the lack of information on preparing for engineering manager interviews**. **The individual interviews were the same for ICs and managers.** **The differences were in the quantity of each segment**. Based on this, **I split my time equally between coding, systems design, and behavioral prep**. Even though only two of my companies required coding interviews for managers, I knew I needed the most help here.

### Assemble the questions

Blind and Reddit offered me a wealth of information on the questions I should study. It all came down to 3 sets of questions:

1. The [Blind 75 coding questions](https://leetcode.com/list/xi4ci4ig/)
2. The 25-35 system designs by [Narendra](https://www.youtube.com/watch?v=umWABit-wbk), [Gaurav](https://www.youtube.com/watch?v=OcUKFIjhKu0), and [Mikhail](https://www.youtube.com/watch?v=bUHFg8CZFws)
3. The behavioral questions on [Exponent](https://www.tryexponent.com/guides/amazon/em-interview/questions) and across the web - about 75 of these as well

### Spread the questions out over time

Since every company has agreed to a 45-minute interview, I knew I could not spend more than that time on any single question.

I also knew that coding exercises were stacked two at a time per interview. That means I would have to cram two questions in a 45-minute coding session and use the remaining time to study any answers that I didn't arrive at myself.

With roughly 175 questions in my study bucket at 45 minutes each, that's 7,875 minutes of study. With a one-year-old and a full-time job on my plate, I knew I couldn't stomach an intense study regimen for more than 3 months. Overall, that's **90 minutes of study per day, every day, for 3 months straight**.

### Finding the time

As a powerlifter, I workout 4-6 days per week, and my workouts often take 60-90 minutes or more. I knew I could at least maintain my progress by cutting down the number of lifting days to three. Stacking exercises back-to-back could create an intense but effective workout in about 45 minutes instead of 90. That reallocated more than two-thirds of my time for studying without negatively affecting anything else, including work, sleep, and family time. I had to find an additional 30 minutes a day.

Once I showed the potential salaries to my wife, we both agreed it would be perfectly reasonable for me to ask for 30 minutes of additional "me time" per day to prepare for my next job,

## Doing the prep

As you can see, even for a dad with a full-time job, I was able to find consistent time to study each day. With my questions, my timeline, and my [study music](https://www.youtube.com/watch?v=1uWj-Up7qT0), it simply came down to picking enough questions a day, every day, for the next 3 months.

I live and die by Asana. I created a project for interview prep and added all 175 questions I wanted to study to one to-do list and scrambled the questions. **Each day my goal was to pick two new questions I hadn't seen before and one old question I hadn't answered perfectly.** All questions that were answered perfectly were completed and never seen again. Any question I messed up was thrown back into the bucket and resurfaced via an Anki deck that timed my spaced repetition.

### How it really went

In truth, this is how I would have liked to implement my study plan from the beginning. In reality, I spent a bunch of time, in the beginning, learning how to prepare. This ate into my study time significantly so I had to play catch up later on.

Further, the behavioral questions were much easier after I wrote down the important milestones of my career. Once I had **two examples for each behavioral question in the [STAR method](https://www.thebalancecareers.com/what-is-the-star-interview-response-technique-2061629)**, the bulk of the behavioral prep was done. Recall often took only a few seconds, so I could easily study 10-20 questions a night.

This allowed me to spend the majority of the second half of the three months on just systems design and coding questions. Spaced repetition allowed me sufficient space to burn these examples into my brain in long-term memory so I wouldn't forget any details going into my interviews.

One thing that really helped minimize the amount of study time on systems design was to listen to the YouTube playlists while I was working out and sleeping. Yes, I would often start a YouTube playlist before bed and fall asleep listening to it. I must have repeated the same 40 videos half a dozen times. I saturated my brain with these concepts so that I could passively retain the knowledge.

## If I could do it again

Someone recently asked me how long I took to prepare and if I felt like it was enough. I think I was slightly under-prepared and would have liked an additional two weeks to study. All things considered, that's a pretty spot-on study plan. And if I hadn't spent so much time trying to figure out how to study in the first place, I easily would have found those extra two weeks to draw from at the beginning of my prep.

Now that I have my Anki decks saved, my playlists on YouTube, and my lists in Leetcode, I could cut my prep in half. I also know that senior managers are not asked coding questions at Google or Facebook. If I were to look elsewhere with a promotion in mind, I would apply a month out. That would provide sufficient time to prepare my behavioral and system design questions with these Anki decks.

## On timing

**Timing is everything in interviewing**. If you're studying for three months, you want to start applying about a month before your prep is over. Companies drag their feet (unless you have other offers, more on that later). **You want to make sure you can stack as many interviews on top of each other**.

For reference, the seven companies I took interviews with all occurred, start to finish, within two weeks. One week for submissions and logistics, two weeks for interview sessions across several rounds, and one week for offers &amp; negotiations.

Doing it this way will not only **save you time** but will **increase your offers** via competing bids. Companies move at a glacial pace when you have no offers. Companies will escalate the interview process, in under 48 hours if you have compelling counteroffers.

### On negotiating

There's not much to say here that hasn't already been said [on this blog](https://haseebq.com/my-ten-rules-for-negotiating-a-job-offer/). Just go read it. Anything I say here would just regurgitate what is written there. It's perfect and describes how I approached my offers.

## So how did I do?

My interview funnel looked like this:

* 16 companies reached out/I applied to
* 7 companies (Amazon, Brex, Facebook, Netflix, Physna, Stripe, YUM) replied and invited me to final rounds
* 2 of 7 companies rejected me
* 1 company I declined before the final round due to logistics. I knew I wasn't excited enough about the role to rush with other offers on the table.
* 1 told me I passed the interviews but never gave me an offer. I suspect it was because the other bids were too high.
* 3 companies gave me offers
* **I ended up at Facebook.**

In terms of negotiating, let's just say for my years of experience, **I would have the third-highest new offer on Levels out of over 200 offers on their site**. And that includes the fact I don't live in SF, Seattle, or NYC where offers are 5% higher than mine with a much higher cost of living.

In the end, **I passed all of my FAANG interviews and landed in the top 2% of offers for someone with my experience**. Because of this, I feel like my interview prep was worth my time and effort.

In the future, I will **ask for referrals for all of my applications** and **not worry about prep**.

Get a referral for all of the jobs you apply for, study a standard list of questions, stack your interviews, and negotiate like a champ. Success is all but guaranteed.
]]></content:encoded>
      <dc:date>2022-03-18T20:53:00+00:00</dc:date>
    </item>
    <item>
      <title>Consistency</title>
      <link>https://www.adamconrad.dev/blog/consistency/</link>
      <description><![CDATA[
  This article is part of the Systems Design Series following the book Designing Data-Intensive Applications by Martin Kleppmann. If you missed the previous article, check that out first.


]]></description>
      <pubDate>Wed, 18 Aug 2021 23:15:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/consistency/</guid>
      <content:encoded><![CDATA[> This article is part of the [Systems Design Series](/tag/systems-design/) following the book [_Designing Data-Intensive Applications_](https://dataintensive.net/) by Martin Kleppmann. If you missed the [previous article](/blog/fault-tolerance/), check that out first.

This chapter, and the last one we will cover in this book series, covers consistency and consensus. It focuses on the <abbr title="Consistency Availability Partition-tolerance">CAP</abbr> theorem. We will explore all levels of consistency guarantees.

## Consistency guarantee levels

There are three grades of consistency guarantees:

* **Weak:** Reads may or may not see writes. Often seen in real-time (VOIP, chat) or in-memory solutions like Memcached.
* **Eventual:** Reads will _eventually_ converge to see the latest write. Data replication is asynchronous. BASE NoSQL databases use this model most often. **Highly available systems** like email use eventual consistency. When convergence occurs is arbitrary and one of the more tenuous aspects of this consistency guarantee.
* **Strong:** Reads always see writes instantly. Data replication is synchronous. ACID relational databases use this model.
  * **Causality:** Something happens after another. Questions precede answers. Causal ordering is often a comparison of a limited set of operations rather than the total ordering of all operations in a system. **This is the practical limit of strict consistency in distributed systems**. Timestamps in a replication log are an easy way to achieve a causally consistent dataset.
  * **Linearizability:** The strictest definition of consistency as it relates to the CAP theorem. Applications do not worry about replicas and maintain the illusion of a copy of the data. It's a recency guarantee that states there is an atomic operation that happens in the database where the data can never revert to the old value after a particular point in time. It is said to evolve data _linearly_. Apache ZooKeeper uses linearizability for leader election to ensure you cannot promote two nodes to be the same leader in [single leader replication](/blog/replication/) schemes.

One final note on linearizability: **single leader replication is your only practical strategy for achieving a highly available system with strong guarantees**. Other replication strategies, such as multi-leader and leaderless replication, utilize asynchronous replication. Because strong consistency requires synchronous replication, we can only rely on single leader replication where the leader handles both reads and writes, and the replicas simply act as fault-tolerant backups.

This limits our use cases for replicated, linearizable systems. Since multi-leader and leaderless replication are out, you cannot distribute your data across multiple data centers. This is because each data center needs to have its own leader.

In theory, you could have multiple data centers that all have to access a central data center where your single leader lives, but if any data centers fail you risk halting all operations. This could crush performance and grind requests to a halt.

The book covers quite a bit more on ordering and linearizability than I would have liked because Kleppmann admits that linearizable consistency is not practical in real-world distributed systems. And yet, there is quite a bit he covers on things like [total order broadcast](https://en.wikipedia.org/wiki/Atomic_broadcast) for such systems that we will likely never need to use.

## On consensus

To have a consistent view of the data across distributed systems, all of those nodes have to agree on the state of the data. This is known as _consensus_. One classical way of achieving consensus is through something called a _two-phase commit_. It's basically like marriage vows: a coordinator (officiant) asks each participant if they are willing to commit to the transaction (marriage). That is phase one, the _preparatory phase_. If every participant says yes ("I do"), then the _commit phase_ executes and the transaction has to complete.

These kinds of algorithms are often implemented in relational databases for distributed systems. You don't see this covered much in NoSQL solutions. And while there are benefits to this system to increase atomicity and durability, there is a great risk for performance. **If the coordinator goes down and the log record of the 2PC is lost, you could potentially lock the database forever**. As we've seen before, we never want a single point of failure. the two-phase commit algorithm rests heavily on the health of the coordinator application to ensure a smooth transaction across a distributed system. How might we mitigate this?

Better consensus algorithms have emerged over the years. The two most commonly-referenced algorithms, both in this book and externally, are [Paxos and Raft](https://blockonomi.com/paxos-raft-consensus-protocols/). They are total order broadcast algorithms so they deliver messages exactly once, in order, to all nodes. In other words, this is a **repeated barrage of consensus decisions for each node to decide on what the current state of the data is**.

A common use case for these algorithms is with [leader election](/blog/replication/). You need to gather a majority vote amongst nodes as to who is the new leader when the old leader dies or goes offline. While this helps ensure that two nodes can't claim to be the leader at once, they do slow down the application significantly. Consensus protocols require synchronous replication of votes which is the slowest way to provide consistency of decisions.

Further, they require a majority voting structure. This means you can't have an even number of nodes since you could have a tie. This also means you can't dynamically add or remove nodes into your cluster. This is because the number of nodes you have at any given moment will have to flip from odd to even. In the end, you have a very rigid and brittle system for consensus that does not support the dynamic nature of modern distributed systems.

### ZooKeeper to the rescue

Can you achieve a fault-tolerant total order broadcast with high performance? [Apache ZooKeeper](https://zookeeper.apache.org/) aims to do just that. Google Chubby, Etcd, and [even Redis](https://en.wikipedia.org/wiki/Distributed_lock_manager#Other_implementations) can act as a distributed lock manager.

All of these systems act as in-memory databases that hold a small amount of information about all nodes. Think of them as an index card of emergency numbers that are distributed to every node. These coordinator services ensure that a tiny amount of information is embedded on every node so that if a decision needs to be made quickly, consensus can be reached without a lot of heavy back-and-forths over the wire.

In fact, tools like ZooKeeper will purposefully make small decisions across a tiny number of nodes to ensure the voting is fast. If a leader dies, for example, it will specifically only target three to five other nodes for a vote, rather than rely on quorum from the entire cluster.

The other benefit of a tool like ZooKeeper is it provides a method for _service discovery_ in partitioned systems as well. It acts sort of like a load balancer in that it can route requests to the correct node or cluster to reach a particular service. In doing so, it can also act as a _heartbeat service_ to check for the membership status of the nodes in its jurisdiction.

If there's anything I've learned over the last several weeks, **ZooKeeper is the swiss army knife of distributed system coordination**. If you're looking for a tool that handles consensus, failure detection, membership, and fault tolerance, this should be the first one for you to grab in your toolchain.

ZooKeeper is not a silver bullet. **Multi-leader and leaderless replication don't require global consensus, and thus don't need ZooKeeper.** As we discussed in a previous chapter, these replication schemes are the most common in distributed systems. These are also the most common schemes to use NoSQL databases. Thus, you won't see ZooKeeper being used outside of distributed RDBMSs.

## Closing thoughts

This wraps up [my series](/tag/systems-design/) on distributed systems following this book. There are quite a few more chapters that are not covered here. I'd encourage you to check out more resources if you're interested:

* [Grokking the system design interview](https://www.educative.io/courses/grokking-the-system-design-interview) is a way to practice the things learned in this series
* [Grokking the advanced system design interview](https://www.educative.io/courses/grokking-adv-system-design-intvw) offers even more practice and more closely resembles the first principles tools used throughout this series
* [This system design primer](https://github.com/donnemartin/system-design-primer) includes flashcards to provide spaced repetition learning of the concepts we've discussed
* [This channel](https://www.youtube.com/channel/UC9vLsnF6QPYuH51njmIooCQ) and [this series](https://www.youtube.com/watch?v=xpDnVSmNFX0&list=PLMCXHnjXnTnvo6alSjVkgxV-VH6EPyvoX) offer an excellent selection of videos on distributed systems design
* [High scalability](https://highscalability.com) is the gold standard for cutting-edge news and research in practical distributed systems. Consider subscribing!

]]></content:encoded>
      <dc:date>2021-08-18T23:15:00+00:00</dc:date>
    </item>
    <item>
      <title>Fault tolerance</title>
      <link>https://www.adamconrad.dev/blog/fault-tolerance/</link>
      <description><![CDATA[
  This article is part of the Systems Design Series following the book Designing Data-Intensive Applications by Martin Kleppmann. If you missed the previous article, check that out first.


]]></description>
      <pubDate>Wed, 04 Aug 2021 00:33:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/fault-tolerance/</guid>
      <content:encoded><![CDATA[> This article is part of the [Systems Design Series](/tag/systems-design/) following the book [_Designing Data-Intensive Applications_](https://dataintensive.net/) by Martin Kleppmann. If you missed the [previous article](/blog/partitioning/), check that out first.

The self-proclaimed pessimistic chapter of the book, chapter 8 covers fault tolerance in distributed systems. Let us briefly introduce why things break in an internet-scale system:

* **Commodity machines break.** Since internet applications are about [scaling out instead of up](/blog/replication/), each node in the cluster is cheap. Cheap things break more frequently.
* **The Internet Protocol is unreliable.** There is no guarantee that things will move as intended. Everything is sent over the wire in _packets_. Traditional telephony networks send everything over in _circuits_. This lack of a reliable, consistent connection presents problems. The request could be _lost_ or _queued_. The node accepting the request could _stop responding_ or _fail_ altogether. Worse, your request could have made it all the way to the node and back but was _lost on the way back to you_ or will arrive _later than you want it to_.
  * **We add in TCP to make it more reliable.** The downside is there are variable network delays since that queuing and reliability slows the system down.
  * **You can trade TCP for UDP if you need more speed.** If you cannot afford delays in data and can afford less reliability, consider using UDP. A Zoom call is a good example of when UDP is a better protocol than TCP.

This is in contrast to vertically-scaled systems that rely on beefy supercomputers. These systems often act as if they were single-computer applications. Therefore, we will not be covering this in the book or in the notes.

The crucial thing to internalize is that **faults happen in the real world all of the time**. The best anecdote from the book is how a shark bit some network cables and took down parts of Amazon EC2. Even a company as powerful as Amazon cannot prevent the almighty shark from screwing up network connectivity.

## How to deal with faults

If we know faults will happen, then it is vital to be proactive in dealing with them.

**Error handling** is a simple and effective way of staying on top of faults. Keeping your customers aware of issues may only require a browser refresh to correct the problem.

**Heartbeats** help detect faulty or down nodes. A _load balancer_ (like HAProxy or Nginx) will periodically ping nodes to ensure they are up to send them traffic.

**Leader election** is a strategy to elect new write replicas when leaders are down in replicated clusters.

**Timeouts are the gate to try all of the above strategies.** For customers, the only way to reliably detect faults is to inform them that too much time has elapsed to process their request. In that time, you will have tried the things above. Once the timeout has been exceeded, it is safe to assume that a fault has occurred.

How do you choose a sensible timeout? **Measure your response times and variability.** Then, choose a number. Unfortunately, **there is no standard timeout interval that will work for everyone.** Apache Cassandra uses the [Phi failure-accrual algorithm](https://www.slideshare.net/srisatish/cache-concurrency-cassandrajvmapachecon2010) to adjust gossip protocols when they check with nodes to determine if they are down. TCP does its own [retransmission](http://blough.ece.gatech.edu/4110/TCPTimers.pdf), but there is not much here for you to take away for your own applications.

### Timing

[Timezones are hard](https://www.freecodecamp.org/news/synchronize-your-software-with-international-customers/). Dealing with time, in general, is difficult. All computers have clocks, but they are rarely in sync with each other. How do you ensure that the time or duration of an activity is accurate?

**Sync them.** Things called <abbr title="Network Time Protocol">NTP</abbr> servers help ensure that the clock you see on your computer is actually correct. If it isn't, the NTP server will send back the correct time and update your clock. The problem is, your clock could hop backward. If you sent a request at `10:00:01` and it was returned at `10:00:00`, you have a problem. That is why these kinds of clocks, called _time-of-day clocks_, are not reliable for measuring time duration.

**Count monotonically.** To safeguard against a standard clock, a _monotonic clock_ is also available on a computer. It is basically a giant timestamp counter that always counts upward, regardless of time syncing issues. You can still use an NTP server to adjust the monotonic clock, but it does not count backward. If there is a syncing issue, you simply _delay when the next value is incremented_. This is generally the preferred method for distributed systems.

Even with both mitigation strategies in place, **no system is perfect**. There is a network delay from the NTP server to your local clock, so they will never fully be in sync. Firewalls, leap seconds, virtual machines, and physical hardware clock malfunctions all contribute to perpetual inaccuracies between the actual time and the time posted on your machine. As long as you accept this premise, you can achieve reasonably high accuracy with the times you use it.

### Strategies around timing issues

With all of these issues around keeping track of time, there are really only 3 solutions that are worth implementing:

* **The last write wins**. Cassandra and Riak use this strategy to determine the most recent write to a database. But this suffers from the same basic problem: _what is recent_? If it is all relative anyway then you could still have two nodes that wrote to the DB with the same timestamp but one node's clock is off.
* **Provide times with a delta**. Google Spanner uses this with its [TrueTime API](https://cloud.google.com/spanner/docs/true-time-external-consistency), but good luck using it outside of Google.
* **Limit certain functions that would cause time delays and pauses**. Garbage collection is a notably slow process that can hang a system. Ensuring there is enough memory to handle other functions when garbage collection grows unwieldy ensures some fault tolerance.

The message you must remember is that **time is relative, and you cannot achieve _real-time_**. Real-time, in the truest sense of the word, is reserved for embedded systems so things like your airbag can deploy when it needs to. Real-time, as it is used for web-scale, is a relative term for feeling instant even if it isn't actually an instant operation.

## Trusting a faulty system

Now that we know systems are unreliable, we need to know that the actors can be trusted. The book assumes they can be and that each player in the distributed system acts with honest intention. This is where the old security saying of **trust but verify** comes into play.

_Quorums_, as we discussed in the [data replication chapter](/blog/replication/), are a way of obtaining votes from all nodes to figure out what the majority believes about the system during a discrepancy.

If you have 3 children and ask, “who stole the cookies from the cookie jar,” you're likely to believe it is child 2 if both 1 and 3 point to 2.

What if one of them is lying? This is the basis for the [Byzantine Generals' problem](https://www.microsoft.com/en-us/research/uploads/prod/2016/12/The-Byzantine-Generals-Problem.pdf). Systems that safeguard against lying actors are _Byzantine fault-tolerant_. They are only needed for mission-critical systems like aerospace or the blockchain. Internet web-scale systems do not require Byzantine fault-tolerance (usually in the form of 2/3rd majority vote rather than a simple majority vote), and we can improve fault tolerance with a few simple tricks:

1. **Require checksums.** TCP and UDP offer these and ensure that corrupted packets must be retried to ensure data is transferred correctly.
2. **Permission, sanitize, validate, error-handle, and output-escape your data.** This [blog post](https://alexkrupp.typepad.com/sensemaking/2021/06/django-for-startup-founders-a-better-software-architecture-for-saas-startups-and-consumer-apps.html) beautifully summarizes a recipe to eliminate bugs from your code when ingesting data. Bugs are a source of lying since the system behaves unexpectedly through faulty code.
3. **Add redundancy to your NTP servers.** Replication of time synchronization server checks ensures that the majority timestamp is the accurate and correct one.
4. **Build systems around crash recovery and partially synchronous system models.** System models take many forms and shapes. One will not see the strictness of a completely synchronous, Byzantine fault-tolerant system in the real world. But you also don't want a fully asynchronous model where even the slightest perturbance shuts down the system. Striking a middle ground with your system model is a sensible and useful real-world approach.

The next chapter will investigate algorithmic approaches to handling real-world fault tolerance with system models that handle crash recovery and strive for partially synchronous updates.

This quote at the end of the chapter summarizes how I would recommend the litany of strategies in this book to a more junior engineer:

> If you can avoid opening Pandora’s box and simply keep things on a single machine, it is generally worth doing so.

It is easy to look at a book like this and think of all the neat technologies you could use on your current project. Resisting the urge to do so and finding ways to say no is arguably a better approach. Most companies and most systems will never require the scale beyond a simple Postgres server. Barring safeguards around data replication, a simple RDBMS can scale to millions and millions of records without much else to power it.

Until we meet again in the next chapter, here is some additional reading material I enjoyed with this week's chapter:

* [A system design question around the "top-k" heavy hitters](https://www.youtube.com/watch?v=iJLL-KPqBpM).
* If you didn't watch any of the previous episodes from this channel, you'll also want to study their implementation of a [distributed messaging queue](https://www.youtube.com/watch?v=iJLL-KPqBpM).
* [How to beat the CAP theorem](http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html) is the inspiration for the system design in the first link.
* [This article on the Kappa architecture](https://www.oreilly.com/radar/questioning-the-lambda-architecture/) challenges the merits of the Lambda architecture with a simpler model that uses fewer frameworks and drops batch processing as a requirement.
* [Comparing modern stream processing frameworks](https://www.youtube.com/watch?v=ZWez6hOpirY) is a great video that helped me make sense of way too many Apache streaming systems and why they all seemed to be redundant upon first glance.
]]></content:encoded>
      <dc:date>2021-08-04T00:33:00+00:00</dc:date>
    </item>
    <item>
      <title>Partitioning</title>
      <link>https://www.adamconrad.dev/blog/partitioning/</link>
      <description><![CDATA[
  This article is part of the Systems Design Series following the book Designing Data-Intensive Applications by Martin Kleppmann. If you missed the previous article, check that out first.


]]></description>
      <pubDate>Wed, 28 Jul 2021 09:00:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/partitioning/</guid>
      <content:encoded><![CDATA[> This article is part of the [Systems Design Series](/tag/systems-design/) following the book [_Designing Data-Intensive Applications_](https://dataintensive.net/) by Martin Kleppmann. If you missed the [previous article](/blog/replication/), check that out first.

This chapter is another central tenant of distributed systems: partitioning. Also known as sharding, we will explore how to split up large data sets into logical chunks.

## Strategies for partitioning

If _replication_ is about copying all data to different databases and datacenters, then _partitioning_ is about slicing up that data across those databases and datacenters.

There are a few ways you can split up that data:

* **By key.** The key can split the rows of data. Imagine 26,000 names evenly split alphabetically. You could put 1,000 entries in a database for the _A_ user names, 1,000 more for the _B_ names, and so on.
* **By hashed key.** You could hash the same key and then split up the data that way as well. The easiest way to ensure information is evenly split is to use a modulo (`%`) operator (hint: just because it is easy doesn't mean it is helpful). If you run the modulo over the number of datacenters, you will always have a key mapped into a datacenter ID.
  * **Add randomness for active keys.** The above assumes that all keys are read and written in an even distribution. For the ones that are not, you can break up keys further with prefix or suffix IDs. Now the same key can be segmented across shards.
* **By a secondary key.** Like the prefix/suffix keys, you can use an additional key with a secondary index. This will make searching faster for the data that is partitioned.
  * **For local document search.** You could make this secondary index the key with the value being the original primary key that you added into the shard in the first place.
  * **For global document search.** Also called a _term index_, you could segment the secondary keys and grab all of the keys across all shards (and shard the term). This is how full-text search databases work.

## Strategies for rebalancing

Over time, these partitioning strategies will skew unevenly. The easiest way to visualize this is to imagine the old hardcover encyclopedias. They are not 26 volumes split by letter. Instead, letters likes `Q` and `X` are combined while `A` and `S` get their own book.

While you may have an even balance of data to start, it will not necessarily evolve that way. You will need methods to rebalance your data from time to time.

Remember how I suggested that we use the modulo operator to segment our keys against the datacenters? This works until you add more datacenters. Then your modulo value needs to change, and you need to rebalance _all_ of your data which creates a lot of rewriting as your data scales. This is not ideal. Here are a few better ways:

* **With a fixed number of partitions.** Rather than use the physical shards as segments, create arbitrary partitions within each shard node. That way, as your shards grow or shrink, the number of partitions stays the same. Then, you simply adjust how many partitions you allocate to each node. _Riak and ElasticSearch use this method._
* **With a dynamic number of partitions.** The previous example is sensible but suffers from the same problem as earlier that you have data evolve in very skewed portions. If your partitioning segments are chosen poorly, your data will be lopsided. Dynamic rebalancing is using something like a B-Tree to change the structure as partitions are added and removed. The data evolves, and so too do the partitions. _HBase and MongoDB use this method._
* **With a proportional number of partitions to nodes.** This is somewhat of a blended approach. The number of partitions is not strictly fixed. It changes as the number of nodes grows and shrinks. This is not the same as dynamic rebalancing because the dynamism reflects solely on the number of nodes and not on arbitrary boundaries that you define. _Cassandra uses this method._

These strategies all depend on the configurations you make as a developer. You can choose how you want to deploy this rebalancing: _manually_ or _automatically_.

With manual rebalancing, a developer assigns partitions to nodes. This requires more effort and time, but it allows you to respond to the needs of your data. Conversely, automatic rebalancing enables the software to do this work for you. Tools like Couchbase and Riak suggest this approach (with approval), but the downside is the unpredictable and possibly suboptimal choices for segmenting and rebalancing your data. A wrong move by the software could be slow and costly.

## How to find data

With all of this data split up amongst so many physical computers, how do you go about finding them?

* **You can keep track of all of them on the client.** The obvious solution is to keep track of each shard in the same way that you have to have your encyclopedias face out on your bookshelf. You have to see which letter you want to access when you approach your bookshelf. The downside is that this is tedious and must be reconfigured every time you add or remove nodes.
* **You can have your request routed for you.** A middle layer can interpret your query or mutation and figure out where your data is for you. This is like in the old days of telephones when operators connected you to the person on the other end. You did not need to know their phone number. You just needed to remember the number of the operator so they could route the call for you.
* **You can throw a request to any node, and that node will route it.** This is like calling up a corporation and going through their touchtone service. You know you want to call up Dell customer support, so all calls get routed to a 1-800 number, but it is up to the service to find the exact extension for you. The default number _might_ get you who you are looking for if you know your party's extension number, but you may have to navigate the tree of options.

Tools like [Apache ZooKeeper](https://zookeeper.apache.org/) handle this configuration management in many popular databases like HBase and Kafka. Engines like Riak and Cassandra use a [gossip protocol](https://en.wikipedia.org/wiki/Gossip_protocol) to chain requests from node to node until the data is found.

---

Partitioning is straightforward and not nearly as dense as [replication](/blog/replication/). There are only so many ways you can split up data. In the [first post of this series](/blog/scalability-reliability-maintainability/), I noted I would be skipping the next chapter on transactions. The last sentence of the chapter explains why:

> In this chapter, we explored ideas and algorithms mostly in the context of a database running on a single machine. Transactions in distributed databases open a new set of difficult challenges, which we’ll discuss in the next two chapters.

Since I care about educating this audience on distributed systems and systems design, it seems only fair to focus on chapters that tackle transactions in a distributed nature.

]]></content:encoded>
      <dc:date>2021-07-28T09:00:00+00:00</dc:date>
    </item>
    <item>
      <title>Replication</title>
      <link>https://www.adamconrad.dev/blog/replication/</link>
      <description><![CDATA[
  This article is part of the Systems Design Series following the book Designing Data-Intensive Applications by Martin Kleppmann. If you missed the previous article, check that out first.


]]></description>
      <pubDate>Tue, 20 Jul 2021 23:30:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/replication/</guid>
      <content:encoded><![CDATA[> This article is part of the [Systems Design Series](/tag/systems-design/) following the book [_Designing Data-Intensive Applications_](https://dataintensive.net/) by Martin Kleppmann. If you missed the [previous article](/blog/storage-and-retrieval/), check that out first.

This chapter is the real introduction to distributed systems. We are shifting into the second part of the book here. You will notice that I intentionally skipped chapter 4 based on a suggested curriculum I outlined in the [first post of this series](/blog/scalability-reliability-maintainability/).

## Horizontal versus vertical scaling

Before the chapter gets started, Kleppmann briefly discusses how we've collectively arrived at a _shared-nothing architecture_ of horizontal scaling. Distributed systems design focuses on scalability so it is important to know the 3 main types of scaling you'll see in real-world systems:

1. **Vertical scaling.** Also known as _shared-memory architectures_ or _scaling up_, this involves adding more RAM, CPU, and resources to a single machine. The key observation is that **doubling resources will likely more than double your costs**. As such, vertical scaling is recommended when heavy computation is required at the expense of fault tolerance and reliability. There is a variant of this called _shared-disk architecture_ which pertains mostly to data warehouse applications that provide additional fault tolerance with multiple machines accessing the same disks on a centralized network. This still suffers from the same issues when you colocate all data in one data center but has limited use.
2. **Horizontal scaling.** Also known as _shared-nothing architectures_ or _scaling out_, this refers to adding more machines with commodity resources. This is the primary strategy used when companies scale out their services and hence is the primary focus of this section of the book.

## Replication 101

Why do we replicate?

1. **Reduce latency.** Keeping data close to your users reduces the time it takes to physically travel over the wire to your customers.
2. **Increase availability.** The server goes down? You've got a backup on a different machine and in a separate data center.
3. **Increase throughput.** Drive-thru windows at your bank are convenient but if the line is long you know there are lots of bank tellers available inside to help you out and keep customers and productive.

This chapter discusses three primary reasons _why_ we need to replicate. We'll offer strategies to resolve why **data changes all of the time and how we deal with it**.

If data never changed then replication would be easy. Since data grows and evolves over time, we employ various strategies based on our system's specific needs.

### Leader replication

The first two methods involve **leader replication**. Each replica, or copy, has a role: `leader` or `follower`. Some schemes invoke a single leader while others leverage multiple leaders. The responsibilities of each are straightforward:

1. **Leaders ingest write data.** Only the leader can modify the database.
2. **Followers copy changes from the leader.** Followers must wait for a log file from the leader with changes they will make in the same order the leader made them.
3. **Anyone can offer read data.** All replicas can return data to a customer that asks for it. This is called _read-scaling architecture_.

That last step is the interesting one. What if you ask for data from a replica that has not copied all of the latest changes yet? This is where replication strategies offer tradeoffs in their approach.

From here, you can focus on two ways of getting changes to replicas:

1. **Synchronous replication.** As the name implies, you send changes to followers immediately. You then only inform customers when all changes are completed successfully. Practically, this doesn't work because if a follower replica hangs then the change fails which hurts durability. To be fair, **synchronous replication is the best strategy to eliminate the risk of data loss**. In reality, when systems use synchronous replication, they will follow a hybrid _semi-synchronous replication_ strategy where one follower is synchronous while the others are asynchronous. Read-scaling architectures don't work with this scheme because followers have to wait for leaders to synchronize data which keeps them too busy to accept additional reads.
2. **Asynchronous replication.** In this scheme, changes are sent to followers immediately. The difference is you can inform customers when _some_ changes complete successfully. You are not beholden to every follower sending a successful response. This is the most common scheme in production, high-scale application. This trades off speed for no guarantees in durability and eventual consistency.

**What if you want to add a new follower to the gang?**

1. **Periodically take snapshots of the data.** This is good anyway for backups. Send the latest backup to your new follower.
2. **Poll the leader again for changes since the snapshot.** Now you can assume the new follower is asking for changes as if this were an asynchronous replication strategy. Each database engine has its own scheme for issuing catch-ups such as log sequence numbers or binlog coordinates.

**What if things fail?**

* **If the follower fails** then you execute the same catchup strategy for a new follower. Just read from the logs the last changes since the failure and then play catchup.
* **If the leader fails** then you have to elect a new leader. You can either do this _manually_ or _automatically_. Manual promotion benefits from careful failover to a verified follower. The downside is that you have to invest developer time into failovers and can be cumbersome if they occur with some frequency. Automatic promotion relies on selecting a new leader, usually with the closest representation to the leader's data.

Reaching consensus with the automatic strategy is fraught with problems:

* **There might be a gap in the data, even with the best follower chosen.** In this case, you violate strict durability and you incur data loss which is never good.
* **Multiple followers might believe they are the leader.** This can also lead to data loss if both start accepting writes to propagate to the other followers.
* **The heartbeat interval may be too long and miss the latest changes.** Even if you have one follower with a perfect copy of the leader's data, the leader may have been down long enough to miss some new writes. If the heartbeat to check for its health is in a long enough interval then new changes could have been requested while the leader was down which will be lost for good.
* **The old leader may not relinquish their throne**. Assuming everything else was correct, you still might see a variant on the second issue where the old leader starts back up and still thinks it is the leader. Now you are in a similar scenario where two leaders are consuming writes and you incur data loss.

Logs help with restoring the data in addition to helping new followers join in on the fun. Each database engine utilizes one of four replication log schemes:

1. **Statement-based.** These copy raw SQL commands. They are simple but require deterministic commands, which can be too strict for most databases.
2. **Write-ahead.** These are logs written before changes to the main disk are stored. This is a common append-only log that is used with things like [SSTables and B-trees](/blog/storage-and-retrieval/) but requires all engines to follow the same format. If the format changes with breaking changes to an engine, then it will require downtime to upgrade. This is what plagues things like major version upgrades in tools like Postgres.
3. **Row-based.** This focuses on the values of rows instead of the commands like the previous two. This decouples the data from the storage engine used, which solves the problems from the previous strategy. New versions of Postgres use this, also known as _logical log replication_. The book doesn't seem to see a downside to this approach but, [this Stack Overflow thread](https://stackoverflow.com/questions/33621906/difference-between-stream-replication-and-logical-replication) goes into this strategy in greater detail which I appreciated. In short, things take more time because one database is updated at a time methodically.
4. **Trigger-based.** This strategy leverages SQL triggers and stored procedures. While this benefits from functionality native to the language instead of the database engine, these triggers can be complex. Further, this log strategy does not work for NoSQL solutions or any database that doesn't leverage SQL, such as graph databases.

#### Eventual consistency

As mentioned earlier, a read-scaling architecture can provide additional performance for applications that are heavy on reads. The downside of this strategy is that you may read outdated information from a follower node. When a leader node sends data asynchronously to a follower node, it cannot wait to know if the follower replication succeeded.

This is what is known as _eventual consistency_. At some point, _eventually_, the data will be synced across all replicas. There is no guarantee when that eventual time will be. But, it is meant to be a reasonably short amount of time so that there is no real compromise in reliability. Nonetheless, there are occasional complications with eventual consistency:

* **Reading right after writing.** Say you update your email. If you make those changes and refresh the page then you'll want to see your new email address saved. _Read-after-write consistency_ is a strategy around this but is complicated in distributed systems.
* **Reading into the past.** Say you write a few checks to your bank account. Your bank account starts at $5, then you add two checks of $5 each. In theory, if the eventual consistency is delayed enough between replicas, the leader could show $15, one replica could show $10, and yet another replica could show the original $5. _Monotonic reads_ ensure you only read from one replica so this problem doesn't occur. Of course, if the replica fails, you're back to square one.
* **Answers to questions you haven't asked.** In the previous scenario, you could see a later state before you see a previous state. Not only are you going into the past but you are messing up the causality of the data - you wouldn't expect an addition to lead to subtraction of the balance. _Consistent prefix reads_ to ensure that the writes happen in a certain order. If you have a sharded database, you can't guarantee this fix unless all of the data is affixed to the same shard. This becomes more complicated but does solve the issue.

All of these problems are difficult to solve with a single-leader replication strategy. They become less difficult when you invoke a multi-leader replication strategy.

#### Multi-leader replication

As the name implies, multi-leader replication allows you to assign multiple leaders at once. This solves nearly all of the problems you see above:

* **No need for manual promotion.** With multiple leaders, you don't have to worry about being without a leader at all. Restarting leaders assumes there is at least one leader already able to receive writes.
* **No gap in writes.** Unless you get incredibly unlucky, a leader will exist to ensure there is no downtime of a leader to accept writes. This can happen with a single leader. With multiple leaders, you guarantee there is always a place to accept writes.
* **No leader conflict resolution.** If you know who the leaders are, you can always create a backup scheme so you either fall back to an alternative leader _or_ you have a simple election to keep things smooth.
* **Heartbeat intervals no longer matter (to a point).** Waiting a long time to see if the leader is still online is not a problem because you have multiple leaders to fall back on. As long as their heartbeats aren't synced and are checked in a staggered fashion, you can ensure that you're constantly aware of at least one online leader.
* **Multiple leaders don't necessarily require follower promotion at all.** One problem with a single-leader scheme is the leader won't give up its role as leader when it comes back online. If you have other leaders available, there is no need to promote them. You always have leaders available to do all the work you need. If a particular leader goes offline, you just wait for it to return and push the additional load to your remaining leaders.

In addition to solving problems from single-leader replication, there are a few use cases where multi-leader replication really shines:

* **Datacenter replication.** Have multiple data centers? Assign a leader for each data center to ensure all locations have write coverage. The challenge is that you are back to single-leader replication per data center if you don't elect multiple leaders per site. Further, writes become far more complicated since you have multiple data centers that have to resolve data changes. That requires costly RPC calls to copy logs from one data center to the other.
* **Offline applications.** What if your mobile phone were a data center? When you're offline, you store changes to your local database. When you get back online, you sync your changes with a leader at a data center for persistence and backups. This is multi-leader replication taken to the extreme where every device acts like a leader.
* **Collaborative applications.** Tools like Google Docs allow you to edit concurrently with other users. Think of your browser as a leader where you publish changes locally and see them instantly in the browser. In the background, you take those changes and push them to a remote leader replica. This replica syncs your changes with the changes of another individual. They ensure they all match nicely.

Everything sounds great - so what's the catch? It should be obvious at this point, but the big issue with multi-leader replication is **write conflicts**. With multiple leaders comes multiple nodes where you can write changes to your database. Who has the correct answer? Is there a leader of leaders?

The easiest way to mitigate write conflicts is to simply **avoid having them in the first place**. This may seem like a cop-out answer, but you can design your systems toward conflict avoidance. Having one leader per data center and routing users to specific data centers is a way to avoid conflicts. Specific updates in a particular range of data go to dedicated leaders. Of course, this reverts to being a microcosm of single-leader replication, so if a data center goes down, you are forced to reconcile concurrent writes on different leaders.

If you can't avoid conflicts, you can use some _tie-breaker heuristic_ such as assigning UUIDs to data centers or records so that the highest numbered ID wins. Simple, but prone to data loss. In theory, this also allows you to merge conflicting values, but then you'll need some sort of user intervention to reconcile the concatenated data.

Of course, you could implement your own custom conflict resolution scheme. Amazon has done with this some surprising effects, such as having items reappear in users' shopping carts after they have purchased items. While a suboptimal user experience, it is still likely better than losing data.

How do leaders communicate with each other? The book identifies three specific _topologies_ for communication, and I see there are only two:

* **Every leader talks to every other leader.** This ensures everyone gets the message from everyone else with a direct line of communication between every leader. The problem here is there is no causality to the writes because there is no order with which leaders are updated. As opposed to...
* **Leaders update each other in a chain.** An example of this would be a circular or star-shaped topology. In these schemes, there is an ordering to how leaders communicate with each other which solves the causality problem. The challenge is that if a replica fails during the update chain, it breaks the chain, and all downstream updates will stall out. If a leader is early enough in the chain, most other leaders could suffer from data loss and stale information.

As mentioned before, the best course of action is generally to avoid conflicts in the first place. Many database systems support some kind of conflict resolution scheme, but none are perfect.

### Leaderless replication

The last major strategy with replication is leaderless replication, where **every node can accept reads and writes.** Amazon Dynamo is the most famous example of leaderless replication and has ushered in a resurgence of this strategy. Cassandra, Riak, and Voldemort have also adopted leaderless replication schemes. This scheme is ideal for applications that require **high availability and low latency** at the expense of a very loose definition of **eventual consistency and stale reads**.

When every database replica can write data, you open yourself up to lots of staleness. There are two primary strategies for dealing with this stale data:

1. **Use subsequent reads to detect stale data.** The next time a user reads data, read it from multiple replicas. If any of those are stale you can issue follow-up writes to ensure the stale nodes are brought back up to the latest versions of the data. This is commonly called _read repair_.
2. **Run a heartbeat to detect stale data.** Every update comes with a version number. Run a background process that pings every node to see what version of the data it has. If any of the nodes are less than the latest version number, update the databases. Continue to ping all databases in regular intervals to ensure staleness is mitigated promptly.

Leaderless replication schemes often implement _quorums_ on reads and writes to create consensus when there is a discrepancy. This is a tuneable threshold that allows developers to configure at exactly which point a vote passes to reconcile when replicas argue over which version of the data is the latest.

**Quorums really only work if you have sufficient replica nodes to break a tie.** This requires votes from read replicas, write replicas, and ensure those votes outnumber the number of replicas that debate the values. As long you have a majority vote on the state of reads and writes, you can proceed with quorum voting.

Further, **monitoring is difficult with leaderless replication**. This is because there is no universal order to how data is consumed. If a node goes down, or worse, several nodes go down, you risk destroying the quorum.

A concept known as **sloppy quorums** was designed to mitigate against this. It states it is better to **write the data even if a quorum is not achieved** when the required nodes for a quorum vote are down or offline. They still ask for the designated number of votes, but they may not come from the approved pool of original nodes that belong to the quorum. Without the approved set of nodes, you have fewer guarantees into the latest state of the data since these nodes are not as closely monitored.

This method _increases write availability_ but require further mitigation like _hinted handoff_ to restore the quorum node team. It also requires backup plans like read repair to ensure that restored nodes eventually receive the latest updates.

Another issue you can run into with leaderless replication is **concurrent writes**. A single change is propagated across multiple replicas. Each replica is subject to its own network latency and IO. There is no guarantee that any write will arrive at all replicas at the exact same time. How do you reconcile this case?

**Last write wins** is a simple strategy that dictates that the latest write is the one that will persist, and all other writes for the same data are considered stale duplicates that can be dropped. This sounds great in theory, but in practice, this is _bad for durability_.

Say your bank account reads $1. If you somehow issued 2 concurrent updates, mistakenly adding $2 instead of $4, you may run into trouble. Both updates will say they were completed successfully. But if the $2 change arrives after the $4 change, your bank account will update to $3 even if that isn't ultimately the correct answer. Cassandra mitigates this with a UUID attached to every write update to ensure seemingly concurrent operations are indeed differentiated.

There are numerous algorithms discussed in the book to determine concurrency. **Concurrency is not concerned with things happening at the same _time_ but happening _without knowing about each other_**. It's extremely difficult to say with certainty that two events occurred at the _exact_ same time. But they could occur in a close enough interval to not know about each other based on the state of the application.

Riak knows how to merge concurrent values with its _sibling merge_ functionality. Riak also uses  _version vectors_ to issue read repair so that clients can retrieve them on reads and return them back to the databases on writes. My takeaway here is if you are looking for **robust leaderless replication, look no further than Riak**.

---

That wraps up the chapter on replicating distributed data. Another way to distribute data is through partitioning. This is the topic of the next chapter so stay tuned!
]]></content:encoded>
      <dc:date>2021-07-20T23:30:00+00:00</dc:date>
    </item>
    <item>
      <title>Storage and retrieval</title>
      <link>https://www.adamconrad.dev/blog/storage-and-retrieval/</link>
      <description><![CDATA[
  This article is part of the Systems Design Series following the book Designing Data-Intensive Applications by Martin Kleppmann. If you missed the previous article, check that out first.


]]></description>
      <pubDate>Mon, 12 Jul 2021 22:33:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/storage-and-retrieval/</guid>
      <content:encoded><![CDATA[> This article is part of the [Systems Design Series](/tag/systems-design/) following the book [_Designing Data-Intensive Applications_](https://dataintensive.net/) by Martin Kleppmann. If you missed the [previous article](/blog/data-models-and-query-languages/), check that out first.

This chapter is all about getting and sending data. There are two main kinds of data stores:

1. <abbr title="Online Transaction Processing">OLTP</abbr>. This includes your traditional relational and NoSQL databases. They comfortably store terabytes of data, are highly available, and are designed for the end-users of your applications.
2. <abbr title="Online Analytical Processing">OLAP</abbr>. This includes data warehouses from names like Oracle and IBM Db2. They store into the petabytes of data, are primarily read-only copies of several OLTP databases, and are designed for business analysts.

One thing to note about OLAP data stores is the simplicity of their data models. While the entire [last chapter](/blog/data-models-and-query-languages/) was dedicated to the variety of OLTP data models, there are really only two models in use for OLAP systems:

1. _Star schemas_, named after the star-like shape representing the central fact table that points out to supporting dimension tables
2. _Snowflake schemas_, which add on sub-dimensional tables to the above star schema with a more complex, snowflake-like pattern

The seemingly obvious comparison is that snowflake schemas are more complex because of their additional normalization with sub-dimensional tables. Conversely, star schemas are simpler to conceptualize with less branching. Either way, both support lots (100+) of columns.

## Column-store databases for big data

Wide-column fact tables benefit from column-store databases. As the name suggests, you're inverting the way data is stored and retrieved. Rather than store data in rows, where each entry corresponds to a single object and various properties of that object in columns, you store all of the values for a single property together.

Since a massively wide fact table is likely only utilizing several columns in a given query, data organized by column lends itself well to these kinds of analytics queries. You can partition and shard data by column which makes searching against particular columns even more efficient.

Since all data in a given column adheres to the same type, you can think of a column as one giant array. This makes column-store databases effective at compressing data with techniques such as bitmap encoding. With more stuff able to fit in memory, column stores can effectively use computer caches to leverage lots of CPU cycles to iterate through data. You can further improve efficiency when grouping or filtering by sorting your columnar data.

Finally, column stores can leverage highly specialized materialized views called data cubes. Data cubes make it easy to reference a lot of aggregated data. This is a way to precompute data and display it for fast retrieval.

All of these make reads incredibly efficient for even the largest data sets. The clear downside is the cost to write data. A table with 500 columns might now be spread out across 500 files, one for each column. That means adding a new entry means writing 500 properties to 500 files. If that entry just happens to max out the file size on disk you'll have to perform even more writes to partition and segment out your data. In other words, writing is expensive. As we'll see later on in this post, there are tools like LSM-Trees that can effectively speed up data storage.

## Data retrieval techniques

Storing data is straightforward. Write data to an append-only log and save the file to disk.

Retrieving data, on the other hand, that's where things get interesting. The book presents a progressively complex way of retrieving data efficiently.

### Hash index

_Indexing_ is the general strategy for retrieving data quickly. By writing only the data you search against into memory you trade space for speed. The downside, besides the additional space requirements, is the cost to write the data. Data stored in multiple places requires multiple writes to disk and takes more time.

One way to index is with a hash code, just as hash maps/tables relate keys to values. An example would be Bitcask. Bitcask is the storage engine powering Riak.

As we saw from the [YouTube](https://www.youtube.com/watch?v=bUHFg8CZFws) example, we need to count video views. The video offers Cassandra as the data storage mechanism of choice. The book might suggest that Riak with Bitcask might be the ideal choice. Bitcask uses an in-memory hash table separate for what is stored in disk. This, in turn, makes frequent writes still fast on read.

Everything about this sounds great, right? We have a method that is:

* _Scalable_: we can shard the data on disk while keeping the indexes in memory
* _Fault-tolerant_: the database can go down in a separate data center that stores the index
* _Eventually consistent_: indexes can now be updated independently of the data stored on disk
* _High throughput_: the right hashing strategy limits collisions which allow for a lot of conflict-free storage

There are two limitations with hash indexes:

1. _Slow range queries on indexes._ Since the index key is a code and the codes are unique you don't gain any speed searching a range of values.
2. _Size limitations._ If you run out of memory the whole thing falls apart. Extreme-scale applications that fill a hash table index completely are the one exception to scalability.

### SSTable

The next evolution of hash indexes is Sorted String Tables, also called SSTables. SSTables provide some additional benefits:

* They only store keys once, saving on storage space.
* They sort the keys to further speed lookups.
* They use a variant of merge sort to merge and compact segments.
* The organization of that data can be implemented as a red-black or AVL tree, allowing you to write in any order and read in sorted order. This makes it easier to store in memory.

I didn't touch on segments earlier. A _segment_ is just **a means of sectioning off a data log**. To prevent running out of space you need a means of separating your indexes across files. To reduce the space requirements we _compact_ the segments by removing duplicate keys and only keeping the most recent write to that key.

This implementation of an in-memory balanced tree for keeping a cascading tier of SSTables is aptly called a **Log-Structured Merge-Tree** or LSM-Tree for short. Lucene, the [Apache search library](https://lucene.apache.org/) that powers ElasticSearch and Solr, uses something like an LSM-Tree to store its dictionary of terms and SSTable-like files to map those terms to postings.

More specifically, if you're dealing with a dictionary of words you'd want to use a structure that tailors itself to strings instead of numbers. We've already evolved past a hash table so that is out. A self-balancing binary tree is out, too. Suffix/prefix trees like [tries](https://en.wikipedia.org/wiki/Trie) are specific to strings but suffer from the same problem of taking up lots of space and eating away at RAM. So what's left? That's where the mighty [Bloom filter](https://en.wikipedia.org/wiki/Bloom_filter) comes to the rescue.

A Bloom filter is a probabilistic data structure that is sort of an inversion of other data structures. Rather than using the information to find if data is contained within a structure, a Bloom filter tries to determine if something is definitely not in the set or if it might be in a set. The "might" part here is key because it's not guaranteeing validity. There is a chance for a false positive.

With a sufficient number of hashes, you can obtain a low false-positive rate to obtain extremely accurate results with a very low amount of indexing space. This solves our biggest problem with indexes when the data is really large and you're worried you will run out of in-memory storage for your index. A Bloom filter can handle extreme scale and still perform well with the tradeoff that your results aren't always 100% accurate.

### B-tree

We've talked about efficiently retrieving data in memory. What if you want to retrieve data on disk? The [B-tree](https://en.wikipedia.org/wiki/B-tree) is the most widely-used and heavily studied database index lookup tree. I had to build one of these for my relational database class in college. It’s another self-balancing tree structure used in all of your major RDBMSs like Postgres and MySQL.

While both LSM-Trees and B-trees contain key-value pairs sorted by key, the strategy for separating data is completely different. The Log-structured indexes can be of variable size. They can also be quite large, spanning into multiple megabytes. B-tree indexes, by contrast, are always fixed blocks of typically 4 KB or more. Since disk pages are also aligned in fixed blocks, the B-tree is better suited for sorting and retrieving data on disk instead of in memory.

B-trees are structured in layers so that the lower levels represent the range of values in between two values in the parent layer. As we saw earlier, in-memory indexes suffer from poor performance in range queries. B-trees do not.

The _branching factor_ determines how wide each layer of the tree is before the range must be split down to a lower layer. For conventional databases with these 4 KB pages, you could have a branching factor of only 500 and only need to traverse down 4 levels before you have stored 256 _terabytes_ of data. That's a flat and efficient tree!

In terms of system design, how does this compare in performance to the in-memory solutions?

* It is also scalable because the partitions are small, 4 KB pages.
* The data structure inherently supports range queries as a result of the branching factor.
* Fault tolerance is achieved through a write-ahead log which writes the data changes to the WAL before it is written to the B-tree (hence, write-_ahead_).
* Locks on the B-tree called latches can be introduced to preserve consistency in an ACID-compliant manner, achieving a higher level of consistency than with a log-structured index.

When comparing log-structured against B-tree indexes, the main rule of thumb is to **consider performance on reads or writes**. Though performance testing will confirm this for your implementation, log-structured indexes are thought to be better for writes. Conversely, B-trees are considered better for reads. This is because LSM-Trees frequently write, segment, and compact their SSTables to create efficiency.

On the other hand, B-trees offer a more reliable indexing scheme because the keys are only written once and do not suffer from a high level of writes due to compaction. The pages are separated into small, predictable pages and do not require sizable computation for write compaction and segmentation. This is particularly useful for transactional databases when ACID guarantees are a must. That would explain why B-trees have stood the test of time for so long.

### Other indexes

The book offers a few more indexing suggestions, such as secondary indexes for joins, clustered indexes where the value is a heap file rather than the raw row data, multi-column indexes like [R-trees](https://en.wikipedia.org/wiki/R-tree), and fuzzy-search indexes like those we talk about with ElasticSearch and Solr.

Then some indexes are colocated with the whole database in memory, such as the case with Memcached or Redis. Now the entire thing, both the index and the raw data, are stored in memory. You only need a hard disk for fault tolerance and durability. Further, in-memory databases allow for more flexibility in the data structures they utilize. It is not uncommon to interface with a priority queue or set with something like Redis.
]]></content:encoded>
      <dc:date>2021-07-12T22:33:00+00:00</dc:date>
    </item>
    <item>
      <title>Data models and query languages</title>
      <link>https://www.adamconrad.dev/blog/data-models-and-query-languages/</link>
      <description><![CDATA[
  This article is part of the Systems Design Series following the book Designing Data-Intensive Applications by Martin Kleppmann. If you missed the previous article, check that out first.


]]></description>
      <pubDate>Thu, 08 Jul 2021 22:33:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/data-models-and-query-languages/</guid>
      <content:encoded><![CDATA[> This article is part of the [Systems Design Series](/tag/systems-design/) following the book [_Designing Data-Intensive Applications_](https://dataintensive.net/) by Martin Kleppmann. If you missed the [previous article](/blog/scalability-reliability-maintainability/), check that out first.

This chapter focuses on how you can organize and retrieve data from a database. It primarily focuses on the three primary data modeling paradigms:

1. Relational
2. Document
3. Graph

## Relational model

This is the incumbent. The relation is the _table_. The individual datum within the table is the tuple called the _row_. Properties of a table exist on the _column_. You know, like any other table you would see in the real world.

I grew up with these database systems. They are what I studied in college. My master's thesis was on <abbr title="Relational Database Management System">RDBMS</abbr>. They have been the dominant force in data management for most of the last 40 years.

The most cited open-source examples of relational databases are **MySQL** and **Postgres**. They're still useful and are the first choice for many frameworks like Ruby on Rails or Django. It's just that they suffer from a few problems:

* **They cannot scale without considerable effort.** They are much harder to replicate or partition than they should be.
* **Limited expressiveness.** Querying and organizing relational schemas fits a specific mold. You have to be okay with that or else you'll have to seek alternatives.

So what are the alternatives?

## Document model and the rise of NoSQL

NoSQL (which people would now call "Not Only SQL") describes any kind of database that doesn't follow the relational model. One thing I want to highlight is the term _polyglot persistence_. This refers to the idea that **you can have both relational and nonrelational databases working together**. It's not like you have to choose one paradigm over the other. In fact, it might be advantageous to use both for your application to serve different needs.

I think this is important to call out because NoSQL implies that you're specifically _not_ choosing SQL. It's as though they can't work together in an application but in fact, they can and do.

* something like a resume could be a series of joins between a user and their job experiences OR you could nest those joins into one cohesive JSON model
* just one thing to fetch
  * it actually looks like the thing you're representing (a resume _is_ a document) - also means many-to-one and many-to-many relationships don't work so well
  * fast: requires no joins since it is self-contained (also means poor support for combining disjointed data)
  * highly flexible: no federated schema (also means no referential integrity)

### How do you know the document model is right for you?

Are you working with documents like profiles, resumes, HTML pages? It's great. If you can populate that whole document it's very efficient. If you have a lot of nested relationships or links to other documents it starts to become cumbersome.

A list of jobs might be two separate tables in a relational model. In the document model, they will all be listed out. The exception is if you have created a separate document for all job types. At that point, you're heading in the wrong direction.

One-to-many, tree-like data structure? Good. Many-to-many, graph-like data structure? Bad - stick with a relational or graph database.

So when does something look like a document in the real world? The two often-cited examples I see are:

* **CMS**. The primary object of a blog engine is a blog post which is a document. The important thing is that it is _unstructured_. The content of the blog doesn't need to fit a particular format.
* **E-commerce product catalog**. Imagine a service like Amazon. They have _tons_ of products. Other than the SKU there is really no mandatory format for structuring how you display a product. This falls into the _large volumes_ of _unstructured_ data category which could be a good fit for a document model database.

### Mongo is to Python as Postgres is to Java

This analogy just kept swimming in my brain. One of the key benefits (and drawbacks) of a document database is its loose constraints on schemas. The book makes an interesting analogy in that it's like runtime compilation: no schemas are validated on writes on the way into the database but you can validate a schema on read. This means you can only be assured of the structure of the document when it is being read from the database.

#### SQL is to CSS as JSON is to the DOM

In a similar but not nearly as clean analogy, we can further think of the paradigms of database programming languages. Relational databases use declarative SQL like CSS. They optimize around _patterns_ rather than _algorithms_ because you just _declare_ how you want to filter out your data and you let the language choose the algorithm. This allows for terse, parallelizable code where the language can handle performance optimization and you focus on just getting your data.

Since NoSQL databases don't necessarily use SQL their method of storage and retrieval revolves around their documents, commonly stored in formats like XML and JSON. In theory, you can use any old imperative programming language like JavaScript to interact with the database. This is nice in theory since you are programming UIs and business logic in the same programming paradigm. The downside is that your code is likely more verbose, uglier, and requires a specific algorithmic sequence to execute. Think of it like styling a page with JavaScript; you could - but why would you do that to yourself?

### Writing is not my strength...

Reads are the name of the game here. **Small, unmodified documents are best.** Anything else you risk heavy performance penalties. Documents get rewritten as a whole so minor updates to big documents are a huge drain on performance. The benefit of having all of the data shoved into one document is to remove the need for complicated things like joins. The drawback is without joins you have to colocate data. This can become expensive if the document becomes expensive.

### ...and it may not matter

Postgres supports JSON columns. Mongo supports foreign references in a weird sort of join operation. The truth is, the document and relational models may not be all that different soon so you aren't locked into one idiom between the major database providers anymore. The book goes into an example of this with the MapReduce extension on MongoDB. I didn't think the example was worth stating here but the last sentence on this topic is a great summary of what I took away from this section:

> The moral of the story is that a NoSQL system may find itself accidentally reinventing SQL, albeit in disguise

## Graph models

Remember how earlier I said one-to-many relationships were a good fit for document databases and many-to-many were good for graph or relational databases? Now let's pick that apart further.

If your many-to-many relationship is simple, you can probably get away with a relational database. **For complex many-to-many relationships, a graph database is a good choice.**

What's an example of a complex many-to-many relationship? You already know canonical examples:

* **Social networks.** People know lots of other people. Just ask [Kevin Bacon](https://en.wikipedia.org/wiki/Six_Degrees_of_Kevin_Bacon).
* **The web.** Web crawlers are designed to demystify the tangled mess of connecting and indexing every website.
* **Transportation.** Roads, railroads, and airports all connect points across the globe. This is a [hard problem](https://online.stanford.edu/courses/soe-ycs0012-shortest-paths-revisited-np-complete-problems-and-what-do-about-them) and one I've [explored before](/blog/np-completeness/).

To be honest, I found this section to be a bit dense on specific kinds of graph databases that may not be relevant today so I'll gloss over each section with the critical parts I took away.

### Property graphs and Cypher

[Neo4j](https://neo4j.com/) is _the_ example I see all over the web on graph databases. I think if you just knew this one and [Amazon Neptune](https://aws.amazon.com/neptune/) (which seems to support all kinds of graph database models) you'd be fine.

**This is how I would imagine you would design a graph data structure at scale**. There are two main objects in a property graph DB: the `vertices` and the `edges`. Like a database, each of those things has a UUID. They also have information as to where they go and who uses them. And as the _property_ name suggests, each of these objects also has metadata in the form of key-value pairs.

The power here is in the _flexibility to build graphs_ and the _tagging you can store to find your vertices and edges_. Property graphs don't restrict how you construct your graph. **Graphs don't require a schema other than the basic vertex and edge**. So you're free to design it however you like. The power of the model is in _how you relate things_.

These key-value tags make it easy to find information in the graph _without ruining your data model_. It also means **you don't have a traverse a graph to find information**. Tags and IDs allow you to quickly lookup vertices or edges without having to go through all of the graph connections. They're like the shortcuts to traveling across the network.

**If you need to evolve your features it doesn't ruin the data**. The key-value pairs allow you to create new associations between your core objects without your graph fundamentally having to change. Sounds like Facebook would be a good example of this: people are your edges and their connections are the vertices. In the old days of Facebook, you could list out your favorite bands or movies. A property graph database would allow you to add something like TV shows without having to totally screw up your existing graph or its associations.

**Cypher is to property graphs as SQL is to relations**. That's all you need to know here. It's a declarative programming language for Neo4j in the same way SQL is a programming language for databases like MySQL, Postgres, MariaDB, and SQLite.

### Triple-store graphs and SPARQL

This is another graph database variant. Instead of vertices and edges as the primary objects a triple-store graph uses, you guessed it, three objects: `subjects`, `predicates`, and `objects`. This one intuitively makes less sense to me than a property graph because I don't think of a graph as I think of an English sentence.

Here is how you make sense of these three things:

* **Subjects are always vertices**. They are nouns and stuff like _people_ and _fruit_.
* **Objects and predicates have 1 of 2 states**.
  1. **The object is another vertex**. Then the _predicate is the edge_ connecting `subject` and `object` together.
  2. **The object is a primitive type**. Then the _predicate is a property_ where the `object` is a data type of the `subject`.

The book lists [Datomic](https://www.datomic.com/) as an example of a triple-store graph but it doesn't market itself as a graph database but an amalgamation of many flexible data models. For practical purposes of this audience, you’re likely not to run into a scenario like this.

**SPARQL is to triple-store graphs as Cypher is to property graphs**. SPARQL is the programming language for triple-store graph databases using the [RDF data model for the semantic web](https://www.w3.org/RDF/). Given that the Semantic Web never really took off, it's more of a reason why you don't need to invest more than a few paragraphs into understanding these things.

## Further reading and study

Since this was a much meatier chapter, I found I had some additional notes from the previous [walk-through](https://www.youtube.com/watch?v=bUHFg8CZFws) that was relevant here. There is another video [in that same series](https://www.youtube.com/watch?v=iJLL-KPqBpM) I used for practice this week.

I found myself referencing a few articles which may be helpful in understanding NoSQL databases:

* https://shopify.engineering/five-common-data-stores-usage - Shopify did a nice job of simply explaining the use cases for five different kinds of data stores. The segmentation is a bit odd here and it doesn't align with what I've read in the book but it's still factual information even if the categorization is faulty.
* https://www.prisma.io/dataguide/intro/comparing-database-types#relational-databases-working-with-tables-as-a-standard-solution-to-organize-well-structured-data - This Prisma article was great because it expanded my knowledge even further beyond NoSQL databases to NewSQL, multi-model, time series, and more.
* https://blog.nahurst.com/visual-guide-to-nosql-systems - I like that this post has a diagram that revolves around the <abbr title="Consistency Availability Partition tolerance">CAP</abbr> theorem. It helped me place where NewSQL fits and how to actually tell the difference between HBase and Cassandra.
* https://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis - Another mega list of NoSQL solutions which categorizes based on popularity rather than storage type. I actually really liked this because it simplified my thought process to a pretty rock solid list:

1. **Want relational?** Use Postgres.
2. **Want an in-memory key-value store?** Use Redis.
3. **Want relational without the schema?** Use MongoDB.
4. **Want web analytics?** Use Cassandra.
5. **Want a full-text search?** Use ElasticSearch.
6. **Working with documents, specifically a CMS or CRM?** Use CouchDB.
7. **Building a search engine or log analysis?** Use HBase.
8. **Working with graphs?** Use Neo4J or OrientDB.

### System design tradeoffs between SQL and NoSQL solutions

A few questions you have to ask yourself when comparing SQL and NoSQL solutions as they relate to scalability:

#### How do you scale reads? Writes?

For SQL solutions, you'll want to shard servers when the database runs out of space. Use a cluster proxy to abstract away the shards. Use a configuration service like ZooKeeper to ensure all shards are online and be sure to toggle backups in the event of a downed shard.

For NoSQL solutions, you'll also want to perform sharing. However, there is no need for a cluster proxy because each shard knows about the other. Thus, no configuration service is needed.

#### How do you make reads/writes fast?

For SQL solutions, you'll want to use a shard proxy to cache results and publish metrics, and terminate long-running processes.

For NoSQL solutions, no shard proxies are required because all nodes know about each other. Quorum nodes are used to ensure you don’t need to talk to every node to get a consensus on whether or not the operations succeed.

#### How do you prevent data loss?

For both SQL and NoSQL solutions, you'll want to create read replicas of shards in different data centers.

#### How do you achieve consistency?

For SQL solutions, you'll want to sync lead data to follower replica data. This leads to consistent data. It also takes a long time to process.

For NoSQL solutions, data is pushed to the read replicas asynchronously. This provides eventual consistency and is fast.

#### Other questions worth asking

* **How do you recover data in case of an outage?** Data recovery is much harder for a relational system that sacrifices partition tolerance so you'll need to focus more on cold storage backups while NoSQL solutions are much easier to replicate their replicas across centers.
* **How do you ensure the data is secure?** I only saw mentions of Accumulo and Riak specializing in security. Otherwise, every database needs to worry about this regardless of storage type.
* **How extensible is the DB to an evolving data model?** Evolving data models are better suited for a graph database than a document database.
* **How easy is it to run on/off-prem?** This question was asked in the systems design video from last week but I don't think most companies really need to worry about on or off-premises solutions anymore.
* **How much will it cost?** Most solutions these days are open-source. The cost will come down to your strategy for your horizontal scaling and partitioning.
]]></content:encoded>
      <dc:date>2021-07-08T22:33:00+00:00</dc:date>
    </item>
    <item>
      <title>Scalability, Reliability, and Maintainability</title>
      <link>https://www.adamconrad.dev/blog/scalability-reliability-maintainability/</link>
      <description><![CDATA[Like my series on Algorithms, I’ve decided that I need to really understand systems design.

]]></description>
      <pubDate>Mon, 28 Jun 2021 15:28:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/scalability-reliability-maintainability/</guid>
      <content:encoded><![CDATA[Like my series on [Algorithms](/blog/why-hiring-is-broken-and-how-im-dealing-with-it/), I've decided that I need to _really_ understand systems design.

I run hiring for multiple teams. Some of our questions revolve around systems design. How can I possibly ask these questions if I couldn't answer them perfectly myself?

It's only fair that I master these concepts. The book that everyone seems to agree with is the best book on this topic is [_Designing Data-Intensive Applications_](https://dataintensive.net/) by Martin Kleppmann.

In the spirit of my last educational series, I will transcribe my raw notes over the next 7 weeks or so. I do this both to teach my readers and also to reinforce the concepts myself. As I recall from my course on [learning how to learn](https://www.coursera.org/learn/learning-how-to-learn), one of the best ways to learn is to teach others.

## The syllabus

I've decided to craft a syllabus for myself as if I were both the teacher and the student to help keep me on track and reinforcing the concepts. If you have a similar learning plan or have questions about mine, please feel free to [reach out](/newsletter).

* Week 1 (this week)
  * Read Chapter 1: Scalability, Reliability, and Maintainability
  * Practice: [Design a scalable system to count events](https://www.youtube.com/watch?v=bUHFg8CZFws)
  * Explore, at a high-level, data tools in the space
* Week 2
  * Read Chapter 2: Data Models & Query Languages
  * Practice: [Top K Problem](https://www.youtube.com/watch?v=kx-XDoPjoHw)
* Week 3
  * Read Chapter 3: Storage & Retrieval
  * Practice: [Distributed cache](https://www.youtube.com/watch?v=iuqZvajTOyA)
* Week 4
  * Read Chapter 5: Replication
  * Practice: [Rate limiter](https://www.youtube.com/watch?v=FU4WlwfS3G0)
* Week 5
  * Read Chapter 6: Partitioning
  * Practice: [Notification service](https://www.youtube.com/watch?v=bBTPZ9NdSk8)
* Week 6
  * Read Chapter 8: Faults & Reliability
  * Practice: [Distributed messaging queue](https://www.youtube.com/watch?v=iJLL-KPqBpM)
  * Read: [Facebook's messaging queue with HBase](http://highscalability.com/blog/2010/11/16/facebooks-new-real-time-messaging-system-hbase-to-store-135.html)
* Week 7
  * Read Chapter 9: Consistency & Consensus
  * Practice: Read the [High](http://highscalability.com/blog/2011/12/19/how-twitter-stores-250-million-tweets-a-day-using-mysql.html) [Scalability](http://highscalability.com/scaling-twitter-making-twitter-10000-percent-faster) [Series](http://highscalability.com/blog/2013/7/8/the-architecture-twitter-uses-to-deal-with-150m-active-users.html) on Twitter and implement it
* Week 8
  * Skim Ch 10 & 11: Stream and Batch Processing
  * More possible practice that leverage stream/batch processing like [Uber](https://www.youtube.com/watch?v=umWABit-wbk)
  * Listen: a [podcast review](https://www.youtube.com/watch?v=grGqCuTcu50) of the book

## Notes on Chapter 1

Chapter one is really an overview chapter. It's easy to read and easy to skim. There aren't too many deep concepts here. I found lots of shortlists/rules of thumb to be all you need to focus on the basics.

### Data storage

When thinking of data storage, consider these 5 buckets:

1. DB engines
  * **Row:** good for <abbr title="Online Transaction Processing">OLTP</abbr>, random reads, random writes, and write-heavy interactions (Postgres, MySQL)
  * **Column:** good for <abbr title="Online Analytical Processing">OLAP</abbr>, heavy reads on few columns, and few writes (HBase, Cassandra, Vertica, Bigtable)
  * **Document:** good for data models with one-to-many relationships such as a <abbr title="Content Management System">CMS</abbr> like a blog or video platform, or when you're cataloging things like e-commerce products (MongoDB, CouchDB)
  * **Graph:** good for data models with many-to-many relationships such as fraud detection between financial and purchase transactions or recommendation engines that associate many relationships to recommend products (Neo4J, Amazon Neptune)
  * **Key-Value:** essentially a giant hash table/dictionary, good for in-memory session storage or caching e-commerce shopping cart data (Redis, DynamoDB)
2. **Search indexes** (ElasticSearch, Solr)
3. **Caches** (Memcached, Redis)
4. **Batch processing frameworks:** good if data delay can be several hours or more and you can store all the data and process it later (Storm, Flink, Hadoop)
5. **Stream processing frameworks:** good if data delay can only be several minutes and you need to store data in aggregate while processing data on-the-fly (Kafka, Samza, Flink)

### Systems design in a nutshell

Approaching systems design falls into 5 steps:

1. **Functional requirements** are designed to get us to think in APIs where we translate sentences into verbs for function names and nouns for input and return values.
2. **Non-functional requirements** describe system qualities like scalability, availability, performance, consistency, durability, maintainability, and cost.
3. **High-level designs** present the inbound and outbound data flow of the system.
4. **Detailed designs** are for the specific components you want to focus on. Focus on the technologies you want to use for the data you want to store, transfer, process, and access.
5. **Bottlenecks & tradeoffs** ensure we know how to find the limits of our designs and how we can balance solutions since there is no one singular correct answer to an architecture.

### Requirements gathering

When asking questions regarding a product spec for a large-scale system, focus on these 5 categories of questions:

1. **Users:** Who are they? What do they do with the data?
2. **Scale:** How many requests/sec? Reads or writes? Where is the bottleneck? How many users are we supporting? How often/fast do users need data?
3. **Performance:** When do things need to be returned/confirmed? What are the tolerance and SLAs for constraints?
4. **Cost:** Are we optimizing for development cost (use OSS) or operational/maintenance cost (use cloud services)?
5. **CAP theorem:** Partitioning is something you know you'll have to account for with highly-scalable systems. So it may be easier to ask what is more valuable: consistency or availability?

**If consistency is most important**, consider an <abbr title="Atomicity Consistency Isolation Durability">ACID</abbr> database like Postgres or even a NewSQL database like CockroachDB or Google Spanner. **If availability is most important**, consider a <abbr title="Basic Availability Soft-state Eventual consistency">BASE</abbr> database like an eventually consistent NoSQL solution such as CouchDB, Cassandra, or MongoDB.

Even better is to [use this diagram](https://blog.nahurst.com/visual-guide-to-nosql-systems) to map your concerns onto a pyramid. Given that you can only ever expect 2 of the 3 parts of the CAP theorem to be satisfied it might actually be better to ask _which property is least important?_ If it's...

* **Consistency** - most NoSQL solutions will work like Cassandra, CouchDB, and Amazon Dynamo
* **Availability** - some NoSQL solutions and some NewSQL solutions like Bigtable, HBase, MongoDB, Google Spanner, and Redis
* **Partition tolerance** - any relational or graph solution like Postgres or Neo4j will work since these are notoriously difficult to partition compared to the other solutions

Though likely everyone [misunderstands the CAP theorem](http://pl.atyp.us/wordpress/?p=2521) so I would read this a few times and internalize the example.

### The three system qualities in 1 line

This chapter can effectively be summarized in 3 sentences:

1. **Scalability** determines if this system can _grow_ with the growth of your product. The best technique for this is _partitioning_.
2. **Reliability** determines if this system produces _correct results_ (nearly) each and every time. The best techniques for this are _replication_ and _checkpointing_.
3. **Maintainability** determines if this system can _evolve_ with your team and is easy to understand, write, and extend.

### Further reading and study

As I said before, this is a pretty simple chapter. I also watched [this systems design walkthrough](https://www.youtube.com/watch?v=bUHFg8CZFws). This video extended these concepts and informed some of these notes. I like to accompany learnings with practice to seed new questions for our own [interview process](https://indigoag.com/join-us#openings).

[This article](http://highscalability.com/youtube-architecture) on YouTube's architecture further reinforces the sample problem on the YouTube video (how meta). You can check your solution against the one that was really used by YouTube.

Finally, you can rifle through a bunch of [these videos](https://www.youtube.com/playlist?list=PLMCXHnjXnTnvo6alSjVkgxV-VH6EPyvoX) fairly quickly as each touches on a small subset of system design techniques.

Check-in next week with a summary of Chapter 2 of the book: _Data Models & Query Languages_!
]]></content:encoded>
      <dc:date>2021-06-28T15:28:00+00:00</dc:date>
    </item>
    <item>
      <title>Effective messaging for candidates</title>
      <link>https://www.adamconrad.dev/blog/effective-messaging-for-candidates/</link>
      <description><![CDATA[Recruiters have told me that they achieve a 7% (at best) success rate for replies to messages from candidates. Meanwhile, I’m able to secure a 60% success rate when I reach out to prospective engineering candidates. The main reason for this is that engineers prefer to hear from other engineers. The weight of a message from a VP or Director is much greater than that of a recruiter or someone outside of the engineering org. That’s why it is so important to message the candidates yourself.

]]></description>
      <pubDate>Wed, 16 Jun 2021 20:28:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/effective-messaging-for-candidates/</guid>
      <content:encoded><![CDATA[Recruiters have told me that they achieve a 7% (at best) success rate for replies to messages from candidates. Meanwhile, I'm able to secure a **60% success rate** when I reach out to prospective engineering candidates. The main reason for this is that **engineers prefer to hear from other engineers**. The weight of a message from a VP or Director is much greater than that of a recruiter or someone outside of the engineering org. That's why it is so important to [message the candidates yourself](/blog/its-not-hard-to-care/).

## How to set up your messages

I reach out to candidates on LinkedIn. It's hard to find candidate emails. Honestly, I'm getting enough of a response that I don't need to stalk them anyway. This saves me a lot of time so I can just focus on the message.

### Start your message off with a real greeting

Don't rely on LinkedIn to create a salutation. Sure, they guarantee you won't screw up the spelling (or worst, copy-paste the wrong name). But they look fake as hell. _Don't be fake_.

### Next, I'll admit they're probably happy where they are

Most candidates are off the market so I'm not going to pretend they're actively looking and excited to see what we have to offer.

_It helps if they have been somewhere for a few years_. I'll call that out too because then they understand that I'm reaching out because it's reasonable to assume they might be starting to look elsewhere. On the flip side, I won't reach out to someone who started a new job 3 months ago. That would just prove I'm not paying attention.

### Why they would be the right fit for the role I have open

After we get the reason for connecting out of the way I connect **what they have on their resume to what I'm looking for**. I'll get specific - their Python experience, a previous employer in the logistics industry, or an interest in microservices and building stream processing tools. Just something to show that I'm reading the fine print and not just the absolute basics like their name.

### Hone the message of why we're a big deal

_We're making software to help farmers raise, sell, and distribute their crops sustainably_. That's it. You need to get your message down to something succinct. It helps that we have a strong mission for folks to connect to (and [I'm hiring](https://indigoag.com/join-us#openings)).

### Close the sale

I always end by saying something to the effect of "if that sounds interesting can we set up a time to chat next week?" This is a lesson I've learned from my days in consulting sales. People are more likely to respond if they **have a specific request with a specific timeframe to respond**. The fewer reasons for a candidate to say no the better.

## Messaging logistics that might be helpful

Now that you're ready to hit send, here are the last few things to keep in mind to further optimize your open and response rate:

* The _best day_ to send a message is **Thursdays**. Or at the least - not weekends and not the start/end of the week.
* The _best time_ to send a message is **noon EST**. It's lunch on the east coast and breakfast on the west coast. People check their emails and messages at this time. If they're already on the platform they have more inertia to read and answer.
* The _best length message_ is **under 500 characters**. I break that rule often. After all, it's hard to capture everything above in under 500 characters. I never go above 600-650 characters. If you can tailor an authentic message in under 500 characters I'd love to see it because I've had a real hard time making that happen.
* **Respond within 24 hours.** I'd advise you to check LinkedIn daily. It helps keep your message congruent in being interested in the candidate by truly wanting to hear from them with a sense of urgency.

With all of these strategies implemented I routinely receive a **60% open rate and a 30% reply rate**. Even if a candidate says no - you can **still win a fan**. Be sure to thank the candidate for opening and responding and be sure to **ask them if they know anyone who would be a better candidate we should talk to**. Most people are willing to offer up a name (or 3). All of these ideas will help further expand your network and get you closer to hiring your next engineer.
]]></content:encoded>
      <dc:date>2021-06-16T20:28:00+00:00</dc:date>
    </item>
    <item>
      <title>How to manage remotely and stay sane</title>
      <link>https://www.adamconrad.dev/blog/how-to-manage-remotely-and-stay-sane/</link>
      <description><![CDATA[The last 14 months have been a grand experiment in how to manage teams remotely. In one sense this has been a great success. Teams have managed to stay productive and companies have shown record profits. On the other hand, there is a concern of an eventual return-to-office culture.

]]></description>
      <pubDate>Mon, 14 Jun 2021 12:34:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/how-to-manage-remotely-and-stay-sane/</guid>
      <content:encoded><![CDATA[The last 14 months have been a grand experiment in how to manage teams remotely. In one sense this has been a great success. Teams have managed to stay productive and companies have shown record profits. On the other hand, there is a concern of an eventual return-to-office culture.

Let's look at a few of the most common issues managers have with managing remote teams.

## Effective communication

Remember these two words: **documentation** and **asynchronous**. Wash, rinse, combine, and repeat. When you communicate with your teams, do so in an asynchronous way; ensure your teammates can view your messages on their time and in their mediums of choice. That means your message will have to be repeated again and again from Slack to email to the wiki.

Speaking of wikis, _if you answer a question, put it in the documentation_. By documenting things in a permanent medium like a wiki, you effectively automate your ability to answer questions without having to repeat yourself. You also create an easy to reference, permanent, searchable document for anyone to access.

The side effect of thinking this way is it **reduces the number of meetings on your calendar**. Pushing towards asynchronous, document-based communication removes the need for meetings on your calendar since you can comment back-and-forth and evolve the document on your own time.

## Video calls

First raised by [the BBC](https://www.bbc.com/worklife/article/20200421-why-zoom-video-chats-are-so-exhausting), the last year has showcased a new phenomenon called "Zoom fatigue." People are sick of seeing little boxes of humans on their laptops and crave real attention from real people. I get that. But that will never solve the problem of your dev who is in Minneapolis and you who are in San Francisco.

First, **level the playing field**. Require that all teammates take calls from their laptops rather than an office camera that projects the entire conference room. This might seem like a trivial/subtle change but it is important for setting context about "us versus them."

You don't want to assume that those who are permanently remote are second-class citizens in your organization. They will feel this way if they see folks in a conference room Zoom and they are on their laptops.

The key difference is in _your attention to others_. If you have a camera right on your face with a laptop in front of you, you tend to focus on the participants in the Zoom. If you have a camera in a conference room you tend to focus on the people in the conference room and the camera becomes background noise. This is good for those in the conference room and bad for those who are remote.

So if you need to abandon the conference camera in favor of all laptops even in the office, you should **consider upgrading your AV hardware**. A recent study suggests that [better audio quality makes you appear smarter](https://tips.ariyh.com/p/good-sound-quality-smarter) so investing in a solid microphone like the Shure SM57 is a great, cheap audio upgrade with an interface like the Focusrite Scarlett 2i2. By extension, getting a better camera (and even lighting) will also improve your perception on Zoom.

Even if you don't have the budget for this at work, it still might be a good investment going forward. If you plan to work remotely for future jobs then these will be investments in your future job prospects as well.

## Whiteboarding

Drawing things on a laptop is notoriously hard. There are few solutions to solving this problem right now. I do have one solution I've found to be useful but it requires investment and practice.

**Buy an iPad with an Apple Pencil.**

I understand this is not a cheap ask. Neither is outfitting a home office with a comfortable, ergonomic chair and desk. However, if you want to commit to working remotely and managing teams then this is a relatively cheap upgrade in favor of a much-improved whiteboarding experience.

I really like the [Microsoft Whiteboard app](https://apps.apple.com/us/app/microsoft-whiteboard/id1352499399) because it is easy to get drawing on the iPad with the Pencil. The other two things I really like about this app are the ability to create concrete shapes from hand-drawn movements and that it is easy to persist old drawings or share on Zoom.

It's not perfect but it's free and I've not yet experienced a limitation using it for work.

## Interviewing

Interviewing remotely is a challenge for sure. You want to create a sense of connection with your candidates.

I've had [this video](https://www.youtube.com/watch?v=DYAUw7NTHJ0) on repeat for the last several weeks because it is jam-packed with advice regardless of whether or not you are conducting interviews remotely. You'll gain a lot of insights on whether or not remote is a limitation for you with hiring.

Specifically to remote interviewing, I think the best thing you can do is **allow the candidate to use their own IDE and share their whole screen**. In other words, use the remote interview as a window into how they would actually do their work on a given day. If you frame an interview with that lens that the question of remote versus in-person hiring is no longer a problem. It is solely focused on providing a great experience and not about whether remote is contributing (or hindering) that experience.

## Strategic work

For some reason, managers believe this is the biggest detriment to remote work. Someone recently commented on a LinkedIn post of mine that "nothing beats getting a bunch of folks together in a room in front of a whiteboard".

People have had to strategize throughout the pandemic. Their companies have done very well financially. While some may prefer in-person to remote I fail to see anything about in-office that is _inherently_ a better experience.

This might be a cop-out answer but I think interweaving many of the suggestions above would solve the strategy problem. Leveraging written communication, improving your setup for video calls, and the right technology makes these kinds of meetings so much easier to express your ideas effectively.

Strategy meetings, to me, revolve around [writing dense documentation](https://lethain.com/strategies-visions/) and may benefit from some whiteboarding with other leaders. A great AV setup with an iPad and a proctor to keep the meeting on track with notes is all you need to have an effective strategy meeting.

## Reading emotions and connecting with people

This last point is perhaps the most difficult to conquer in a remote environment. Body language, by the sheer fact that video cuts out most of the body, is impossible to read remotely. Still, even if you can't see someone's whole body does not mean you can't read emotions or body language at all.

**Not being on Zoom, is, in and of itself, a signal.** Not turning on your video can be a sign of [anxiety or depression](https://www.psychologytoday.com/us/blog/lifetime-connections/202004/dealing-zoom-anxiety), particularly if one usually is on Zoom but has decided not too recently.

If the other party is on Zoom you'll have to be more perceptive about watching visual cues. Again, without obvious body language cues like crossing legs or folding arms, you can still judge emotions through facial expressions. You just have to work harder to find these cues and be more observant. I'm not an FBI agent or a psychologist but there are [books](https://www.amazon.com/Unmasking-Face-Recognizing-Emotions-Expressions/dp/1883536367) on the subject.
]]></content:encoded>
      <dc:date>2021-06-14T12:34:00+00:00</dc:date>
    </item>
    <item>
      <title>Becoming an engineering manager</title>
      <link>https://www.adamconrad.dev/blog/becoming-an-engineering-manager/</link>
      <description><![CDATA[The following is a README I give to all of my new managers. Think of it like a step-by-step guide to the broader 90-day plan you would offer your manager when you start in a new role.

]]></description>
      <pubDate>Wed, 09 Jun 2021 21:00:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/becoming-an-engineering-manager/</guid>
      <content:encoded><![CDATA[The following is a README I give to all of my new managers. Think of it like a step-by-step guide to the broader [90-day plan](/blog/a-90-day-plan-for-managers/) you would offer your manager when you start in a new role.

## Welcome!

Your first few months as a manager here at Indigo are crucial to your success, and we’re here to support you!

This new manager issue is a launchpad, and it connects you with crucial information about being a manager at Indigo. It has been designed to ensure you start your journey as a manager with all the resources and training available.

Items on this issue are identified as either **required** (in bold) or _optional_ (in italics). Optional items may not be applicable to your style or team dynamic. Experiment with them for a few days or weeks. If you discover a useful practice or resources outside of this list, please feel free to leave a comment or edit this file.

### Your First Week

* <input type="checkbox" /> **Schedule a handover meeting with the current team manager**. Items covered should include recent feedback, projects, individual goals, and any shared documents or performance reviews most recently given.
* <input type="checkbox" /> **Schedule 1:1 meetings with team members ([Template](https://getlighthouse.com/blog/one-on-one-meetings-template-great-leaders/)) ([First 1-1 Questions](https://larahogan.me/blog/first-one-on-one-questions/))**
* <input type="checkbox" /> **Update your team and organization wiki pages to reflect the new reporting structure**
* <input type="checkbox" /> **Request access to any team chat channels you should be in**
* <input type="checkbox" /> _Create a README of how you work_
* <input type="checkbox" /> _Read [Transitioning to management as a Technical Lead Manager](https://adamconrad.dev/blog/technical-lead-management/)_

### Your First Month

* <input type="checkbox" /> **Focus on 1-1s: ask questions, build rapport, and discuss career development plans with each direct report ([Questions Reference](https://getlighthouse.com/blog/one-on-one-meeting-questions-great-managers-ask/))**
* <input type="checkbox" /> **Review your hiring process**
* <input type="checkbox" /> _Define or review measurable goals for your management role with your manager in a 1-1_
* <input type="checkbox" /> _Modify this document to incorporate learnings you’ve had as a new engineering manager_

#### Recommended Reading List

- _The Effective Manager_ by Mark Horstman (and the accompanying _Manager Tools_ podcast)
- _Resilient Management_ by Lara Hogan
- _High Output Management_ by Andy Grove
- _The 27 Challenges Managers Face_ by Bruce Tulgan (particularly for conflict resolution)
- _Accelerate_ by Nicole Forsgren, Jez Humble, and Gene Kim
- _An Elegant Puzzle_ by Will Larson
- _Extreme Ownership_ by Jocko Willink (and his Jocko Podcast)
- _The Making of a Manager_ by Julie Zhuo
- _The Manager’s Path_ by Camille Fournier
- _The Talent Code_ by Daniel Coyle

### Days 31-60

* <input type="checkbox" /> _Schedule a coffee chat with your recruiting team_
* <input type="checkbox" /> _Begin incorporating feedback into your 1-1s ([podcast on 1-1s & feedback](https://www.manager-tools.com/2009/10/management-trinity-part-2))_

### Days 61-90

* <input type="checkbox" /> _Begin incorporating coaching & delegation into your 1-1s ([podcast on coaching & delegation](https://www.manager-tools.com/2009/10/management-trinity-part-3))_
* <input type="checkbox" /> _Work with your manager on decision-making techniques to effectively delegate work and get more out of your direct reports ([Template](https://www.groupmap.com/map-templates/urgent-important-matrix/))_

### 3 Months and Beyond

* <input type="checkbox" /> _For Technical Lead or Interim Managers: Decide with your manager if you want to continue down technical leadership (e.g. Staff engineer, Principal engineer) or transition to professional leadership (e.g. Engineering Manager)_
* <input type="checkbox" /> _Walk through your first quarterly goal setting for your directs with your manager ([Template](https://docs.google.com/document/d/1p7-Jo45VAw-RTUK97p6r9G0y0RtbmRxIlRntrmcfwHM/edit#))_
* <input type="checkbox" /> _Learn about the performance review process at your company in your manager 1-1_
]]></content:encoded>
      <dc:date>2021-06-09T21:00:00+00:00</dc:date>
    </item>
    <item>
      <title>How to find great engineers</title>
      <link>https://www.adamconrad.dev/blog/how-to-find-great-engineers/</link>
      <description><![CDATA[One of the hardest jobs as an engineering manager is to hire and build a great team. The other is to retain and engage them (but that’s for another day). The purpose of this article is to walk you through step-by-step how to find and hire great engineering talent.

]]></description>
      <pubDate>Fri, 04 Jun 2021 23:29:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/how-to-find-great-engineers/</guid>
      <content:encoded><![CDATA[One of the hardest jobs as an engineering manager is to hire and [build](/blog/building-teams) a great team. The other is to retain and engage them (but that’s for another day). **The purpose of this article is to walk you through step-by-step how to find and hire great engineering talent**.

## Top of the funnel

* **Create a Monthly Engagement Calendar**. Hiring cycles often happen at the start of the month (and year) so __do most of your sourcing at the beginning of the month__.
* **Reach out to DEVELOPER social networks**. No - not LinkedIn. Not Facebook. Not Instagram. Here’s some that have worked well:
  * _Hacker News “Who is Hiring” thread_. Post every month on the first business day. The challenge is you can only do 1 posting per company. So only have 1 engineering manager post for your company.
  * _Reddit subs for your programming language/framework_. r/technology or r/programming will get you nowhere. But if you’re hiring a Scala developer, r/scala allows you to post job threads.
  * _Slack communities_. Hiring JavaScript developers? There’s a [Slack community for that](https://babeljs.slack.com/).
* **Find language/framework-specific channels**. Still not getting any interest from the first two steps? Here’s an example of where you can source JavaScript developers from:
  * For Node:
    * https://discordapp.com/invite/vUsrbjd
    * https://dev.to/t/node
    * https://hashnode.com/n/nodejs
    * https://spectrum.chat/node?tab=posts
  * For React:
    * https://discordapp.com/invite/0ZcbPKXt5bZjGY5n
    * https://dev.to/t/react
    * https://hashnode.com/n/reactjs
    * https://spectrum.chat/react?tab=posts
* **Do the work yourself**. Your recruiting team, if you're lucky enough to have one, is probably great. And they need your help! Their job is to supplement your hiring, _not do it all for you_.

> The best way to convince developers to join is to be genuinely interested in them and convince them that the role you are hiring for is __perfect for them__.

## First contact

* **Set up a 30-minute phone call ASAP**. Since you’ve posted to so many communities, it can be difficult to respond and check in on them. _Encourage folks to email you to schedule a time to chat on the phone and engage with them personally_.
* **Keep the phone call informal**. You really only need to focus on 4 things:
  * What your company is about
  * What _your team_ is about and how it fits into your company's grand vision.
  * Learn more about the candidate, what they’re looking for, and why they’re interested in you
  * A brief overview of the hiring process (this leads them into the next step by encouraging them to start your interview process).

## Interview unconventionally

You’ve invested a lot of effort into hiring. You may not have implemented all of those strategies but **the most important thing about your interview process should be to stand out from the crowd and wow your candidates with a pleasant & fair interview experience**.

**People _loathe_ contrived interview problems** (e.g. reverse a linked list in memory, traverse a 2D array in a spiral). They especially hate when those problems have nothing to do with the actual work they will be doing. So don’t ask them!

**People _love_ practical questions that test the skills they use**. Interviewing someone on the UI layer? Ask them to implement a mockup. Need an architect? Have them design Twitter. If you engage candidates with problems they will really encounter, you’ll learn their true capabilities and they, in return, will feel properly evaluated (read: happy with the interview experience).

**People are _impressed_ by non-traditional approaches**. I've used tiny take-home pull request exams to engage an often overlooked part of every developer’s core job: reading code. Most interviews require developers to write code, but few ask them to read code.

## Related articles

* https://slack.engineering/refactoring-backend-engineering-hiring-at-slack-b53b1e0e7a3c
* https://angel.co/blog/how-to-hire-world-class-engineers
* https://www.freecodecamp.org/news/hiring-right-an-engineers-perspective-on-tech-recruiting-7ee187ded22d/
* https://triplebyte.com/blog/companies/outcompete-faang-hiring-engineers
* https://www.intercom.com/blog/how-we-hire-engineers-part-1/
* https://www.intercom.com/blog/how-we-hire-engineers-part-2-culture-contribution/
* https://www.intercom.com/blog/inbound-recruiting-engineers/
]]></content:encoded>
      <dc:date>2021-06-04T23:29:00+00:00</dc:date>
    </item>
    <item>
      <title>Job hunting as an engineering manager</title>
      <link>https://www.adamconrad.dev/blog/job-hunting-as-an-engineering-manager/</link>
      <description><![CDATA[I did an AMA on Clubhouse a little while ago and there were a lot of awesome, hard-hitting questions on the panel. So for this last installment, I’m going to write out my thoughts to a the questions and how I answered them. The last question from the panel was:

]]></description>
      <pubDate>Mon, 31 May 2021 11:04:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/job-hunting-as-an-engineering-manager/</guid>
      <content:encoded><![CDATA[I did an AMA on Clubhouse a little while ago and there were a lot of awesome, hard-hitting questions on the panel. So for this last installment, I'm going to write out my thoughts to a the questions and how I answered them. The last question from the panel was:

> How do you decide when and how to leave your company and be confident that you’re making the right decision?

As I've mentioned in a previous article, you don't just wake up one day and say to yourself "I think I'll [leave my job](/blog/how-to-do-an-exit-interview) today!"

It's a long, painful process. It's the culmination of a lot of smaller irritations, frustrations, and setbacks that leave you longing for something more on a greener pasture. And that doesn't even cover the interviews.

Interviewing sucks. I won't deny that. But changing your mindset on [how to approach technical interviews](/blog/why-hiring-is-broken-and-how-im-dealing-with-it) can make all of the difference.

Gergely wrote a [great article](https://blog.pragmaticengineer.com/preparing-for-the-systems-design-and-coding-interviews/) recently on what to study and how to study for the range of engineering management interview questions. It's a very complete and exhaustive list of resources that is the exact same references I would list here so there's no reason to duplicate those efforts. Just read that article and actually do all of the stuff he recommends.

Yes, it will take a long time (like, months). And yes, it is difficult, painful, and arguably a giant waste of time. But as I've told lots of people before: interviewing is a skill just like anything else that you can master.

**All that passing an interview proves is that you are good at interviewing**.

I like to compare the technical interview process to the SAT/ACT/IB standardized exams. The best way to "win" at the SATs is to answer a lot of SAT questions. And all that the SATs really prove is that you're good at the SATs. They [don't predict success in college](https://www.forbes.com/sites/nickmorrison/2020/01/29/its-gpas-not-standardized-tests-that-predict-college-success/) and there's an argument to be made for getting rid of them altogether. See the similarities now?

Regardless, both systems aren't going away any time soon. So you have two choices: don't play the game and limit your options and upside, or play the game even if you don't like the rules.

With companies paying [outlandish salaries now](https://www.levels.fyi) for a field that doesn't require formal credentials or schooling, this could not be a better time to get out there and find your [perfect role](https://indigoag.com/join-us#openings). I would argue that even if you don't like the system we have, you're leaving hundreds of thousands of dollars per year on the table by not playing along.

Study hard, get your name out there, and buy me a coffee when this post convinces you to get that job that's going to make you an extra $80k per year.
]]></content:encoded>
      <dc:date>2021-05-31T11:04:00+00:00</dc:date>
    </item>
    <item>
      <title>It's not hard to care</title>
      <link>https://www.adamconrad.dev/blog/its-not-hard-to-care/</link>
      <description><![CDATA[I handwrite every single email I send to candidates.

]]></description>
      <pubDate>Fri, 21 May 2021 11:04:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/its-not-hard-to-care/</guid>
      <content:encoded><![CDATA[I handwrite every single email I send to candidates.

Every. Single. One.

Of course it doesn't scale.
Of course it is time consuming.
Of course rejection sucks and I'll probably get rejected by 99% of the people I reach out to.

If hiring is your _most important_ activity, then why would you ever put in a "copy and paste"-level of effort for people that could provide MILLIONS of dollars of value for your organization?

Have you ever received an email that was copy pasted? Have you received an email where they used the wrong name (or reversed your first name with your last name)? I can answer yes to all of those.

It's pure lunacy to me that recruiters and hiring managers don't put any effort into the candidates they're trying to hire.

_Always_ write a personal note.
_Always_ pay attention to their experience, their wants, their needs.
_Always_ be a human first, and a hiring manager second.
]]></content:encoded>
      <dc:date>2021-05-21T11:04:00+00:00</dc:date>
    </item>
    <item>
      <title>On business value</title>
      <link>https://www.adamconrad.dev/blog/on-business-value/</link>
      <description><![CDATA[I did an AMA on Clubhouse a little while ago and there were a lot of awesome, hard-hitting questions on the panel. So for the next several weeks, I’m going to write out my thoughts to each of the questions and how I answered them.

]]></description>
      <pubDate>Mon, 17 May 2021 11:04:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/on-business-value/</guid>
      <content:encoded><![CDATA[I did an AMA on Clubhouse a little while ago and there were a lot of awesome, hard-hitting questions on the panel. So for the next several weeks, I'm going to write out my thoughts to each of the questions and how I answered them.

> How do you showcase the business value of the work that your team is doing?

There's something that gives me pause about this question having the adjective `business` in there. If you're building a product, you're adding value to the business. What value is not business value in this context? Or put it another way: **what is valuable that _isn't_ business value for professional code?**.

So I will answer this question with the broader aim to **demonstrate value of the work you do**.

The easiest way to do this is to [showcase your impact](/blog/how-to-demonstrate-impact) on the organization. The best way to automate this is to set up regular OKRs every quarter that relate to business KPIs (i.e. revenue, monthly active users, customer retention). Showing impact with numbers and figures against business targets is the easiest way to bridge the gap with business folks.

The more difficult, laborious way to do this is to compare the value you and your teams have generated against how much [you and your teams' cost](/blog/how-to-justify-your-cost-to-a-client).

Really big companies like Facebook know **how much a developer contributes to the company bottom line** - both in revenue and the number of users they impact. That's a huge draw for some developers: imagine if you alone could make a change that affects _millions_ of people! Now what they don't tell you is that your changes probably amount to altering a button from blue to a slightly more [convincing shade of blue](https://blog.ultimate-uk.com/googles-41-shades-of-blue). So if you want to work on something that will actually move the needle on fixing the planet we live on, [we're hiring](https://indigoag.com/join-us#openings).
]]></content:encoded>
      <dc:date>2021-05-17T11:04:00+00:00</dc:date>
    </item>
    <item>
      <title>Are middle managers useful?</title>
      <link>https://www.adamconrad.dev/blog/are-middle-managers-useful/</link>
      <description><![CDATA[I did an AMA on Clubhouse a little while ago and there were a lot of awesome, hard-hitting questions on the panel. So for the next several weeks, I’m going to write out my thoughts to each of the questions and how I answered them.

]]></description>
      <pubDate>Mon, 10 May 2021 11:04:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/are-middle-managers-useful/</guid>
      <content:encoded><![CDATA[I did an AMA on Clubhouse a little while ago and there were a lot of awesome, hard-hitting questions on the panel. So for the next several weeks, I'm going to write out my thoughts to each of the questions and how I answered them.

> As a middle manager, you have several teams with dedicated managers. If you have good managers, what is your value?

As a middle manager, I hope I'm useful! But I see where this question is coming from: _if you have line managers coaching the developers, what do you need a manager of managers for_?

I think this question suffers from an assumption that there are only two kinds of managers; those that lead people and executives who call the shots.

In reality, there are three kinds of managers and we know this to be true because it exists in the military: junior-grade, field-grade, and general-grade officers. To relate this to engineering management:

* Engineering Manager is a base level manager that runs a team (military company). This is a junior officer (i.e. Lieutenant)
* Senior Engineering Manager is an experienced manager that runs a larger (or multiple) teams/companies. This is a senior officer (i.e. Captain)
* Director is a manager of managers (and thus teams). This is a field-grade officer (i.e. Major)
* Senior Director is an experienced director. This is a senior field-grade officer (i.e. Colonel)
* Vice President manages directors (and thus whole divisions). This is a general-grade officer (i.e. Brigadier General/Rear Admiral)
* Senior Vice President is an experienced VP. This is a senior general-grade officer (i.e. Major General/Lieutenant General/Vice Admiral)
* Executive Vice President is a top VP, and is often an interchangeable with CTO. This is the highest-ranked officer (i.e. General/Admiral)

**Middle managers exist to _set direction_ and _coach line managers_.** For larger organizations, the gap is too large between the executives/generals and the line managers/lieutenants. So there needs to be folks who can relate to the junior managers who aren't so busy building visions and setting budgets with many commas after them.

There are [lots of great books](/blog/teach-yourself-engineering-management) on being a great director. The first two that come to mind that I would recommend are [An Elegant Puzzle](https://www.amazon.com/gp/product/B07QYCHJ7V/ref=x_gr_w_glide_sout?caller=Goodreads&callerLink=https://www.goodreads.com/book/show/45303387-an-elegant-puzzle&tag=x_gr_w_glide_sout-20) and [The Manager's Path](https://www.amazon.com/Managers-Path-Leaders-Navigating-Growth/dp/1491973897). The first one has been my playbook as a director while the second book covers the full manager progression all the way up to VP and has a great section on directors.

Check out both of those books and remember to **coach managers** and **set direction for all of your teams** and you will be extremely valuable to any organization!
]]></content:encoded>
      <dc:date>2021-05-10T11:04:00+00:00</dc:date>
    </item>
    <item>
      <title>On delegation</title>
      <link>https://www.adamconrad.dev/blog/on-delegation/</link>
      <description><![CDATA[I did a recent AMA on Clubhouse and there were a lot of awesome, hard-hitting questions on the panel. So for the next several weeks, I’m going to write out my thoughts to each of the questions and how I answered them.

]]></description>
      <pubDate>Mon, 03 May 2021 11:04:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/on-delegation/</guid>
      <content:encoded><![CDATA[I did a recent AMA on Clubhouse and there were a lot of awesome, hard-hitting questions on the panel. So for the next several weeks, I'm going to write out my thoughts to each of the questions and how I answered them.

> I was recently promoted to an Engineering Manager role. How do I turn off my technical know-how to move to a leadership role?

First of all: congratulations! It's hard to break into the role of engineering manager so great job and welcome to the club!

Second of all: know that you are not alone. This is a _very common_ refrain from new managers.

In fact, one of the most memorable pieces of writing I've ever seen on the topic was an Ask HackerNews thread about switching to management. The very top comment was warning new managers to **delegate more and code less** and how this will not come instinctively to you as a new manager.

It's really easy to revert to what you know; to roll up your sleeves, fire up your terminal and editor, and get to coding the problem when everyone else cannot. **Resist the urge to code whenever possible.** It's extremely counter-intuitive.

You've trained, possibly for your entire career, to solve problems with code. Now I'm telling you to take all of that training and throw it in the trash. How is that possibly a good idea?

I've made this analogy [before](/blog/technical-lead-management) with a whole section on delegation. To summarize: the higher up the food chain you go, the rarer your skill sets become. A CTO can do all of the things you can do but needs to focus on only the things she can do, like hiring executives and setting executive direction and vision for the engineering teams.

By extension, you as an engineering manager now have a set of skills that separate you from your previous self as a developer. Therefore, let your developers do development things because only you can do certain manager things, like hiring, firing, promoting, setting technical direction, and resolving conflicts.

**Focus on the things that make you valuable and differentiated so others can do what they are best at**. It may be uncomfortable at first but it is _necessary_ for your role as you transition into management.

]]></content:encoded>
      <dc:date>2021-05-03T11:04:00+00:00</dc:date>
    </item>
    <item>
      <title>Selling ideas</title>
      <link>https://www.adamconrad.dev/blog/selling-ideas/</link>
      <description><![CDATA[I did an AMA on Clubhouse a little while ago and there were a lot of awesome, hard-hitting questions on the panel. So for the next several weeks, I’m going to write out my thoughts to each of the questions and how I answered them.

]]></description>
      <pubDate>Wed, 28 Apr 2021 11:04:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/selling-ideas/</guid>
      <content:encoded><![CDATA[I did an AMA on Clubhouse a little while ago and there were a lot of awesome, hard-hitting questions on the panel. So for the next several weeks, I'm going to write out my thoughts to each of the questions and how I answered them.

> When you introduce a new project or idea to a team of developers, how do you sell that to the team to get them excited about it?

Believe it or not I used to work in sales. For a little while I ran my own [consulting firm](https://anonconsulting.com) full-time (I still accept clients but for small, one-off consulting reports and advice). Running your own services business is arguably more sales than is it actual service development.

So when folks ask me about sales I point them to the classic book [SPIN Selling](https://www.amazon.com/SPIN-Selling-Neil-Rackham/dp/0070511136). It's for sales people by sales people.

But Adam, I'm an engineering manager. Why would I need to read a book about traditional sales to convince people to buy things?

**Because everything is sales**.

You sell yourself when you interview for a job or go on a date. You sell your products or [services](https://anonconsulting.com) if you've ever been an entrepreneur of a business. And most commonly, you sell _ideas_ to your teammates and stakeholders to convince and influence people to believe in the things [you believe in](/blog/dont-call-it-tech-debt).

So who better to learn sales from than actual sales people? If you ever have the chance to shadow sales people at your current company, **do it**. You will not only build rapport with folks in other departments that you normally wouldn't, you'll also learn a ton about how to influence and persuade people.

**Selling ideas is all about persuasion and influence.** There are books for those as well but I've got a [whole curriculum for that](/blog/teach-yourself-engineering-management/). To get you started, you really only need to remember what the SPIN acronym actually stands for:

* **Situation**: where are you today? What's the current "state of the union"?
* **Problem**: what are the current pain points and frustrations that your new project is aiming to solve?
* **Implication**: what is the value of your solution? What would it look like if these frustrations and problems went away as a result of your solution being implemented?
* **Need-Payoff**: how important or urgent is it for you to solve this problem? What are the benefits of building this solution for customers?

If you can **construct a narrative** that incorporates all 4 of those kinds of questions into the new project or idea you're bringing to your teams then you are much more likely to get them energized and excited to make a real difference for your customers.
]]></content:encoded>
      <dc:date>2021-04-28T11:04:00+00:00</dc:date>
    </item>
    <item>
      <title>Can you grow too quickly?</title>
      <link>https://www.adamconrad.dev/blog/can-you-grow-too-quickly/</link>
      <description><![CDATA[I did an AMA on Clubhouse a little while ago and there were a lot of awesome, hard-hitting questions on the panel. So for the next several weeks, I’m going to write out my thoughts to each of the questions and how I answered them.

]]></description>
      <pubDate>Wed, 21 Apr 2021 11:04:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/can-you-grow-too-quickly/</guid>
      <content:encoded><![CDATA[I did an AMA on Clubhouse a little while ago and there were a lot of awesome, hard-hitting questions on the panel. So for the next several weeks, I'm going to write out my thoughts to each of the questions and how I answered them.

> If you have a small team with exponential growth, how do you make sure that the quick growth doesn’t kill the product or team?

When I think of small engineering teams and hyper-growth companies I think of [Instagram](https://review.firstround.com/how-instagram-co-founder-mike-krieger-took-its-engineering-org-from-0-to-300-people). I've actually been thinking about this company a lot lately.

## A brief aside on premature optimization

We've been struggling with organizational (and architectural) complexity at my job recently. Everything takes too long to build and we have seemingly too many folks tackling a few too many things at once.

One of the ways we were able to convince stakeholders to simplify our architecture (and hopefully simplify our org design) was to reference the story of Instagram's engineering team.

TL,DR: Instagram scaled to 300,000,000 MAUs (monthly active users) and sold to Facebook for $1B with only **six engineers**. That's ludicrous.

That story is now burned into my brain. From this point forward, if anyone tells me we need to scale up or build another microservice I'm going to reference this article on how Instagram could do it on far less.

**So to answer the actual question: the only way I could see a fast-growing, successful product failing is to pre-optimize and scale it before it really needs to**.
]]></content:encoded>
      <dc:date>2021-04-21T11:04:00+00:00</dc:date>
    </item>
    <item>
      <title>On productivity</title>
      <link>https://www.adamconrad.dev/blog/on-productivity/</link>
      <description><![CDATA[I did a recent AMA on Clubhouse and there were a lot of awesome, hard-hitting questions on the panel. So for the next several weeks, I’m going to write out my thoughts to each of the questions and how I answered them.

]]></description>
      <pubDate>Wed, 14 Apr 2021 11:04:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/on-productivity/</guid>
      <content:encoded><![CDATA[I did a recent AMA on Clubhouse and there were a lot of awesome, hard-hitting questions on the panel. So for the next several weeks, I'm going to write out my thoughts to each of the questions and how I answered them.

> As leaders, how do you calculate the productivity of team members?

This one is a slippery slope. I've tried using things like the [DORA metrics](https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance). I've tried more insideous calculations like lines of code, pull requests made, or bugs solved, but those all optimize for the wrong thing: more stuff.

More stuff doesn't make you more productive; it just means you're doing more stuff with your day.

In a recent bout of 1-1s I asked every direct this question:

**Who do you think I'm more likely to promote: the person who spent 2 hours making the company $3MM with a single code change or the person who spent all year working hard to get all of our tests to 100% code coverage?**

Everyone picks the first person because they know the real answer on measuring productivity: **in impact**.

Lines of code, PRs merged, bugs squashed...these are all vanity metrics that are easy to track but don't really measure what you're after.

Instead, encourage folks to write something like a [promotion packet](https://staffeng.com/guides/promo-packets/) on the impact they've made over the last year. A bit of self-reflection will make it easy to see whether or not a person is truly productive by any semblance of an objective measure of productivity.

Any other kind of measure of productivity is a red herring and a bad road to travel down that will only lead to micromanagement and missing the point.
]]></content:encoded>
      <dc:date>2021-04-14T11:04:00+00:00</dc:date>
    </item>
    <item>
      <title>Building teams</title>
      <link>https://www.adamconrad.dev/blog/building-teams/</link>
      <description><![CDATA[I did a recent AMA on Clubhouse and there were a lot of awesome, hard-hitting questions on the panel. So for the next several weeks, I’m going to write out my thoughts to each of the questions and how I answered them.

]]></description>
      <pubDate>Wed, 07 Apr 2021 11:04:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/building-teams/</guid>
      <content:encoded><![CDATA[I did a recent AMA on Clubhouse and there were a lot of awesome, hard-hitting questions on the panel. So for the next several weeks, I'm going to write out my thoughts to each of the questions and how I answered them.

> How do you build teams and how do you differentiate from building a team from 0 to 1 vs. 1 to 100?

Full transparency: I've never hired a first engineer for a company. But I _have_ hired the first engineer for a team.

Hiring the first engineer is helpful if you **other teams they can join.** Even if that team is building something completely different, just having a sense of **camaraderie** is a huge win for people to not feel isolated when they are the first to do something.

It also helps to hire **a more senior engineer** because they are more capable of handling big, ambiguous projects from scratch. All of my new teams start with the most senior person and work their way down. It also makes it easier for newer, more junior developers to learn from more experienced folks when you are onboarding and training people to learn the ways of your new team(s).

Full transparency, deux: I also have never hired 100 engineers for a company either. I've hired 18 engineers from scratch for my teams and referred another 10 to other teams. I would say not much changes after the first squad of 4-6 is built.

Getting those folks is key and once you have that central unit in place it's just a matter of **maintaining and expanding that culture with the same squad blueprint.** So if I'm looking at budget for the next year I try to **never hire a fractional or incomplete squad**.

Obviously for some teams, budgets are too tight to fit into neat buckets like squads. But if you have the luxury of building out full units of teams then be sure to build out a whole and not a partial unit or else you're going to have trouble giving that squad **a central mission and objective to complete.**

Each squad should have a vision and a purpose. If you don't have a complete squad to assemble then you don't have all the pieces you need to accomplish an objective. This will leave folks floundering between teams just to keep resources utilizes on all of the things your folks still don't have time for on their own.
]]></content:encoded>
      <dc:date>2021-04-07T11:04:00+00:00</dc:date>
    </item>
    <item>
      <title>Values</title>
      <link>https://www.adamconrad.dev/blog/values/</link>
      <description><![CDATA[I did a recent AMA on Clubhouse and there were a lot of awesome, hard-hitting questions on the panel. So for the next several weeks, I’m going to write out my thoughts to each of the questions and how I answered them.

]]></description>
      <pubDate>Wed, 31 Mar 2021 11:04:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/values/</guid>
      <content:encoded><![CDATA[I did a recent AMA on Clubhouse and there were a lot of awesome, hard-hitting questions on the panel. So for the next several weeks, I'm going to write out my thoughts to each of the questions and how I answered them.

> What skills (as leaders) do you value the most?

I'd say it's a different answer depending on the kind of person I'm working with. I mostly interact with 4 kinds of people in my day-to-day:

* Developers
* Engineering managers
* Product managers
* Designers

Sure, there are a few fundamental skills that I covet regardless of profession: **problem solving ability**, **clear communication**, **tact**, and **logic** are on the top of that list. There are other things that aren't strictly skills, like kindness and empathy, but they are worth calling out because they are attributes of people I love to work with.

For developers I'd add **coding** and **systems design**. We hire against those skills and software development is such a meritocratic profession that covets our core skills. Respect is held for those that can do any not just say.

For other engineering managers I value **deduction** and **intuition**. Management is more art than science so I really appreciate managers who just _get it_. A lot of that comes from experience and humility. I appreciate all of those things.

A product manager's best skill is **negotiation** and dare I say **charisma**. They have so many stakeholders and so many hats to wear that everyone vies for their attention and priority. A great product manager can make everyone feel like they are the most important person in the room even if they have to make their needs a lower priority.

With designers I'd double down on **empathy**. Much of a designer's job is based on user research and conceptualizing things for other humans. Empathy is even more important if your job is all about researching what your end customer needs, wants, and desires.

For everyone else, resort back to the fundamental skills. Kind, independent thinkers will never go out of style.
]]></content:encoded>
      <dc:date>2021-03-31T11:04:00+00:00</dc:date>
    </item>
    <item>
      <title>You only need to be in two kinds of meetings</title>
      <link>https://www.adamconrad.dev/blog/two-kinds-of-meetings/</link>
      <description><![CDATA[I’ve been on a meeting diet lately. I’ve been rejecting meetings left and right and that’s been pissing off some folks. And you know what? Nothing bad has happened and now I have more free time to work. So how do I do it?

]]></description>
      <pubDate>Wed, 24 Mar 2021 11:04:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/two-kinds-of-meetings/</guid>
      <content:encoded><![CDATA[I've been on a meeting diet lately. I've been rejecting meetings left and right and that's been pissing off some folks. And you know what? Nothing bad has happened and now I have more free time to work. So how do I do it?

## Meeting diet - requirements for a meeting I'll accept

The reason I've rejected so many meetings is because the Outlook invite misses any one of these 4 key pieces of information that I **need** in order to accept an invite:

1. **What is the goal?** It should be clearly stated in a sentence or two (maybe I can skip this if the goal isn't something I can help achieve).
2. **What is the agenda?** I want to know what items will be covered in this meeting (maybe I can attend only a portion of the meeting).
3. **How long is each agenda item?** So many people block off 30 or 60 minute meetings because they are convenient. I want to know _exactly_ how long this meeting needs to be because I'm going to guess it is longer than it needs to be.
4. **Is there any pre-reading I can do in advance?** Is there a wiki document or draft I can comment on? Could I replace the meeting completely with a commenting system or asynchronous communication (most of the time the answer is _yes_).

In addition to this, I'll see **how many people** are on the invite and if any of them are optional. Is everyone _really_ required? Are 40 people in on a meeting that isn't an All-Hands? If so, how can we all possibly contribute to this 30 minute meeting?

Even with everything here I won't double-book commitments and may still need to reject the meeting. But at least with this rubric I can reject with a clear ask (e.g. "can you please attach an agenda so I know what parts of this meeting you need my input on?") that is reasonable and is helpful to everyone involved when the meeting is changed.

## The two kinds of meetings I auto accept

With all of the meetings I reject - there are two kinds I accept without hesitation:

* We need to take collective action
* We're getting a "meeting of the minds" together

Any meeting where you take action is a no-brainer for me: a decision meeting, meetings where we assign tasks after chatting, 1-1s, retros, [planning meetings](/blog/running-effective-planning-meetings). If you ask for a 1-1 with me I'm happy to make time for you.

The other meetings I happily accept are the kind where we have to think out loud together or talk things out. These include all-hands, brainstorming sessions, and 1-1s.

You'll notice 1-1s count in both categories. If you aren't convinced yet that these are your most important meetings you need to [get with the program](https://adamconrad.dev/blog/technical-lead-management/).

Sticking with these two kinds of meetings has cut my meeting time down to only half of my week as a director. At one point I had over 80% of my week in meetings so reclaiming 12 more hours in my week has become a joy.

## Not all conversations are meetings

Last thing I'll say is that I don't consider all interactions with other humans to be considered meetings. As a manager, I think part of my IC work is getting together with potential candidates, doing interviews, and performing hiring outreach.

Yes, I talk to another person and perhaps meet up with them at an event or coffee shop (I miss coffee shops). The key difference is **these meetings don't need agendas, goals, or anything explicit in the calendar**. If we're talking about Indigo to join our company then we both know why we are here and I don't need to subject you to the Meeting Diet regimen. In that case, just enjoy your coffee and talk as humans talked before it all took place through Zoom.

Did I mention I miss coffee shops?
]]></content:encoded>
      <dc:date>2021-03-24T11:04:00+00:00</dc:date>
    </item>
    <item>
      <title>Are exit interviews a waste of time?</title>
      <link>https://www.adamconrad.dev/blog/how-to-do-an-exit-interview/</link>
      <description><![CDATA[Someone I knew recently asked how to perform an exit interview to elicit an honest response from the employee for the company to improve.

]]></description>
      <pubDate>Wed, 17 Mar 2021 11:04:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/how-to-do-an-exit-interview/</guid>
      <content:encoded><![CDATA[Someone I knew recently asked how to perform an exit interview to elicit an honest response from the employee for the company to improve.

The short answer is **don't bother because you're not going to get honest feedback from those that leave**. The incentives just aren't aligned - what could the employee possibly gain by spilling the beans now that they are out the door? At best, there is a cathartic, vindictive release for Finally telling the company why they screwed up. At worst, it burns bridges with people who might hire you in the future at another company.

A longer answer is that people do leave genuine feedback from time to time. For the same reasons, the same lack of incentives for the employee are often the reason they do leave honest feedback. If you have nothing to lose and your gripe with the company is directly squarely at the company and not your manager then you'll probably throw them a bone and say something constructive to help your former manager's team going forward.

Still, most people leave [because of their manager](https://www.prnewswire.com/news-releases/new-ddi-research-57-percent-of-employees-quit-because-of-their-boss-300971506.html) which means most people have no interest in digging themselves a further hole with the person that caused them to leave. For the rest, its still not a compelling reason to say anything.

## You already know the answers

Nobody just wakes up and quits out of the blue. They think about leaving. They _plan_ for months very carefully on how they will leave.

Look at how wild hiring is now. Most folks need to study and prep for months.

So that means at least one foot has been out the door for a while. If you’re [doing regular 1-1s](/blog/technical-lead-management/) you’d pick up on this stuff.

If other people have left whatever they tell you is probably the same as it is today.

People know what is broken with their company. Everyone knows and one person leaving, unless they are very senior, isn’t going to be enough of a shock to rock the boat and radically change direction. Because if you missed 10 whole things you would know by now. And you would have fixed them so you didn’t lose the candidate in the first place.

These things all play out in a very long time frame.

## What if you have to conduct an exit interview?

I would just keep it pleasant and let the employee speak freely. If they have nothing to say that’s probably expected and fine too.

If you have a trusting, safe relationship with the person leaving then you are more likely to get an honest response. **I just wouldn't bank on it ever**.

Instead, see exit interviews as the last chance to leave a positive impression before they leave a review on Glassdoor or Blind. An amicable leave means a great review and possible referrals. Leaving with a bad taste in their mouth will only spell trouble for your team going forward.
]]></content:encoded>
      <dc:date>2021-03-17T11:04:00+00:00</dc:date>
    </item>
    <item>
      <title>Hitting deadlines</title>
      <link>https://www.adamconrad.dev/blog/hitting-deadlines/</link>
      <description><![CDATA[My recent productivity obsession is around Evidence-Based Scheduling.

]]></description>
      <pubDate>Tue, 09 Mar 2021 11:04:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/hitting-deadlines/</guid>
      <content:encoded><![CDATA[My recent productivity obsession is around [Evidence-Based Scheduling](https://www.joelonsoftware.com/2007/10/26/evidence-based-scheduling/).

Software estimation is hard. But I also want to make sure our stakeholders understand how far we are along in our projects and when we think we can reasonably deliver some software.

While watching _The Last Dance_ this weekend about the '98 Chicago Bulls basketball team, I got to thinking about how great sports teams all review tape from their previous performances.

Who does that in software? I don't think I've seen anyone do that. And yet, if you think about the principles behind the book in _Deep Work_, you have to get as much feedback as possible as to how you are performing at your job.

_Practicing_ estimation is a skill and takes effort and time. In speaking with a few colleagues and friends on how they've implemented this themselves, there are a few things to keep in mind.

## Buy-in and messaging is key

On the surface, this sounds like micromanagement to the _n_ -th degree. Now my boss is tracking my time? **I'm not tracking your time - YOU are**. It's really easy for a framework for this to sound like a way to place the lens of Big Brother over your development teams. That is _not_ the goal at all.

Instead, this is about tracking _yourself_ to see how good you are at estimating your own work. Endless articles have demonstrated that we, as an industry, are _really_ bad at estimating how long our work will take. This is putting that into repeated, frequent practice.

There are no points for completing things early and there is no shame in finishing things way off target. The point is not to grade your performance in matching expectations. **The point is to see what is your pattern for estimation in the first place and account for that.**

Maybe you are the chronic optimist and everything you do actually takes 20% longer than you expect.

Great! Now we can ensure that if you take a ticket we can reliably add 20% to the estimate to predict when any of your work will arrive. The time to complete doesn't matter; all that matters is the message to stakeholders about when we think this will most likely arrive.

## It takes a lot of work upfront

A friend of mine mentioned that his team of 8 took about 8 hours _a week_ in meetings breaking down tickets in grooming and providing granular estimates of the work that was to be delivered. That's a _lot_ of time in meetings per week, particularly for individual contributors.

At first, that seems like the effort is way too high for a team size that small. However, after a few weeks of working through this system, they were able to get these meetings down to about 2.5 hours while also increasing their accuracy and delivery.

Would you spend 5% of your week ensuring your delivery is 90% accurate to stakeholders? I'd take that each and every week without fail!

## Target practice

Remember that all of this is not an excuse to let stakeholders pepper your team with deadlines. **It is an anti-pattern to adopt EBS _so you can enforce deadlines_**. Rather, it is a reaction to an anti-pattern of deadline-driven development so you can get your teams out of the corner you've already backed into.

The end result of adopting evidence-based scheduling should look something like a bull's eye target. This analogy I use time and time again with my team leads when one of their stakeholders is adamant about requiring a specific date for something to arrive.

Imagine you are firing an arrow at a target 100 yards away with a 5' radius. The goal, of course, is to hit the bull's eye in as few shots as possible. There are two ways to do this:

1. **Increase your accuracy**. Each time you fire an arrow (i.e. deliver an estimate) it should get closer to your target. In other words, if you think something will take 2 days and you're off by 1 day this week and then off by 2 days next week, your estimation skills are only getting worse.
2. **Move closer to the target**. It's a lot easier to hit a target from 50 yards away than 100. In software, this is just time moving forward. The closer you get to the projected date of delivery (i.e. target) the clearer the bull's eye becomes. You probably know what you hope to accomplish today but who knows what you're confident to deliver 9 months from now.

So if you practice hitting a target regularly (i.e. practicing evidence-based scheduling) while simultaneously moving closer to the target (i.e. building more software every day), you should be moving away from deadlines towards target dates which just get more _accurate_ and _predictable_ over time.

At the beginning of the quarter, your project will arrive in 3 months with 60% confidence. In 2 months the project will arrive in 3 weeks in that month with 80%. In that week, the project will arrive on that Thursday with 95% confidence. This is how we shift away from deadlines to target dates.

Stakeholders still hear a date that makes them happy and you aren't beholden to a rigid delivery date which makes you happy. Everyone wins. That is why I am so excited to try this system out and if you have any experience with EBS please let me know.
]]></content:encoded>
      <dc:date>2021-03-09T11:04:00+00:00</dc:date>
    </item>
    <item>
      <title>On culture fit</title>
      <link>https://www.adamconrad.dev/blog/on-culture-fit/</link>
      <description><![CDATA[I was asked during a recent panel on Clubhouse about assessing culture fit. What was once an innocuous term for “people who aren’t terrible human beings” has quickly become a trigger for exclusivity, gate-keeping, and homogeneity. Here are my thoughts on how things got out of hand with this hiring practice.

]]></description>
      <pubDate>Sun, 28 Feb 2021 11:04:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/on-culture-fit/</guid>
      <content:encoded><![CDATA[I was asked during a recent [panel on Clubhouse](https://www.linkedin.com/events/engineeringmanagementamawithsen6770107754863570944/) about assessing culture fit. What was once an innocuous term for "people who aren't terrible human beings" has quickly become a trigger for exclusivity, gate-keeping, and homogeneity. Here are my thoughts on how things got out of hand with this hiring practice.

## The benefits of finding a cultural fit

I can understand where culture fit is seen as a necessary tactic at some places. At small startups, companies are looking for chemistry between the founders and these early-stage employees. I get it; all of you are going to spend a lot of time together taking a big bet that the thing you are building is going to change the world. I was the first employee at a startup and I spent many nights working with the founders until 4am and it all started with a poker game.

For these early companies, you're looking for employees to **define your culture** which is largely based on the principles that the founders already hold dear. So if you, as founders, have a semblance of culture, you're looking to ensure the keys to your initial success stay intact when expanding to hired employees.

On the surface, this makes sense. Many years later I'm looking back on some of these interviews with these founders and I'm left wondering if the toxic elements germinate even at the smallest of startups.

### A culture of homogeneity

I remember a very peculiar interview I had with an early-stage company. We did quite a few technical rounds followed by a round of beers at a local bar. I knew it was about getting to know the founders but in retrospect, it was a sign of being part of the group.

Was I a techie? Did I wear the right hoodies and jeans? Did I know how to hold down a beer (or five) and still gun sling code like a pro? Was I willing to do whatever it took to get the job done in a work-hard-play-hard atmosphere?

This is where the original idea of finding chemistry ballooned into a flawed practice I see at many companies now, both large and small.

## How to fix the culture fit interview

My answer, by the way, on the panel was to completely do away with traditional culture fit interviews. I think they are a stain on the tech industry that we will look back on in 5-10 years as a "what were we thinking?" kind of clarity. **Culture fit interviews as you see them today promote the opposite of Diversity, Equity, and Inclusion (DEI) - Homogeneity, Inequity, and Exclusion (HIE)**.

### Seek additive perspectives rather than reductive ones

Culture fit can be a tactic to find a *homogenous* group of people. It's a test to determine if people are like you. A common refrain here is "can I see myself having a beer with this person after work?" So ask yourself:

* Why do we need to have beers? Why not tea?
* Why do we need to hang out after work? Isn't a coworker relationship during normal hours enough?
* Why do we need to be out somewhere together? Does that diminish the contributions of my remote colleagues?

Diversity is about different perspectives from different backgrounds. And it's not just about demographics like race and socioeconomic background. The kinds of companies you've worked for, whether big or small, all bring new perspectives about how to run and scale an organization.

The kinds of jobs your team has had in the past contribute as well: "purebred" software engineers from CS schools offer a completely different window into building software than folks who never went to college and spent the last 5 years working in retail who just recently transitioned into a software job.

A good interview asks "**how can this person's perspective _add_ to our existing culture**"? There's no specific question that will give you this answer. You just have to remove the bias out of your assessment which takes a lot of hard work and conscious effort.

### Do the work of getting more people in the door

Traditional culture fit assessments are *inequitable* because it is a subjective evaluation with an unspoken set of rules. There is no way to get a correct answer to a culture-fit interview. That is because every company has a different way to "grade" that question which makes it impossible to provide a way to improve or level the playing field for those that need a boost to balance their technical capabilities and job qualifications.

Meanwhile, equity is directly aligned with providing a boost for those who have been systematically left out of the conversation. If you're used to talking to white MIT grads from Boston or Silicon Valley then **you need to explicitly devote more time and attention to the overlooked BIPOC candidates without the same signaling credentials**.

You're trying to explicitly create room and opportunity for those that are often overlooked. If they already aren't part of the in-group, how do you expect to help them with an in-group culture fit question about grinding exams back at Stanford before a kegger?

### And then do the work of getting them in the club

Finally, culture fit interviews can be *exclusive* because those unwritten rules I spoke of earlier are designed to weed out people who don't fit the mold of the [prototypical](https://www.imdb.com/title/tt0151804/) [engineer](https://www.hbo.com/silicon-valley) or any other kind of employee you're used to seeing. The question is all about showing signals that you're part of the group.

Even without a culture fit question, people will assess culture on first impressions. Have you ever grimaced at an engineer coming into an interview in a suit? Made a snap judgment that the person might be too uptight or corporate? We've all subjected others to unfair biases and it doesn't make things better to put someone through an entire interview centered around this concept.

**Inclusive interviews are about bringing everyone into the conversation without adding more reasons to deny someone a seat at the table**. It's about the opportunity for people in a field that is dominated by white males and has been this way for decades. It is our responsibility and duty to represent and include all kinds of people in a way that better reflects the diversity of the people in our nation and in our world.

So what does a _good_ culture fit interview look like when these principles are introduced?

* **It is open to a wider candidacy pool in the first place.** Step one is making sure this interview is asked to more than just white males. Ensuring you remove bias in your job descriptions and in your phone screen assessments ensures that more candidates from more diverse backgrounds land the on-site interview in the first place so they have the opportunity to answer the culture fit interview at all.
* **It asks the candidate to share how their experience is relevant in solving the problems you are facing.** Could your startup benefit from someone at a big company who can deal with your fast growth and scalability issues even if they've never worked for a small team before?
* **It values the universal qualities of exceptional coworkers.** I'd be hard-pressed to find a hiring manager who wouldn't value adding an employee who works hard, strives to do the right thing for customers, and brings positive, uplifting energy to their coworkers. Can you look for those traits in a wider demographic of candidates?
* **It is evaluated on no right answers.** This is the biggest trap of the culture fit interview today is that there is one right answer that signals for culture fit. Instead, look to see if an unconventional answer complements your culture and adds to it rather than fits in an "acceptable" list of responses.
]]></content:encoded>
      <dc:date>2021-02-28T11:04:00+00:00</dc:date>
    </item>
    <item>
      <title>Things to remember with annual reviews</title>
      <link>https://www.adamconrad.dev/blog/things-to-remember-with-annual-reviews/</link>
      <description><![CDATA[Annual reviews have closed from the new year so we’re now into performance review season. A lot of people I work with seem to have a lot of reservations and anxieties about this time of year. I find that if you keep the following things in mind you tend to have really smooth and easy reviews.

]]></description>
      <pubDate>Mon, 15 Feb 2021 21:54:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/things-to-remember-with-annual-reviews/</guid>
      <content:encoded><![CDATA[Annual reviews have closed from the new year so we're now into performance review season. A lot of people I work with seem to have a lot of reservations and anxieties about this time of year. I find that if you keep the following things in mind you tend to have really smooth and easy reviews.

## Weave performance reviews into your 1-1s

The first thing I say to all of my directs every year is "there shouldn't be anything in here that you don't already know and if there is we should talk about it."

We never talk about it.

That's because we **bake feedback into every weekly 1-1**. Anything my folks are doing well (or need work on) are delivered with no more than a one week delay from the moment the event occurs. This allows us to create a true feedback loop where we see performance improvements several times a year rather than once at the annual review.

In fact, I've never had to write an individual development plan for a direct. That's because we don't construct massive waves of change in performance. We tackle issues as they come up rather than in the constraints of an annual review system.

**Tackle one issue at a time and then move on**. If a recurring theme emerges then that is the time to make note of it in the performance review. But that doesn't change the actual game plan. We still tackle feedback and encouragement in real-time and just make sure to highlight major wins/concerns in the annual review based on the previous year's evolution of that feedback.

## Give folks time to read the review

Don't deliver your review at the same time your directs are reading your feedback for the first time. This creates information asymmetry and an awful feeling for your directs. They not only waste time in the meeting reading the review you wrote them but they also can't even begin to process what they have read.

Imagine you gave a particularly negative review. How do you think they are going to feel when reading that information _in front of you_? Probably some combination of mortified and paralyzed because they not only feel bad but they feel like they can't show it so they have no place to go. And if they are non-reactive you might believe that they don't care or aren't willing to take action on this feedback into the next year. Can you see how this is bad for both sides of the review?

Instead, **give directs at least 24 hours to read their reviews before you meet with them**. A week is even better. Since you've already baked feedback into your weekly 1-1s this should already be an easy read but now you save even more time from your meeting by pushing off their reading time out of your scheduled meeting so at a bare minimum you just gained some time back in your calendar.

Now you can use most of the time for your review to **discuss goals for next year and how your feedback can be incorporated into those goals**. This brings me to my last point...

## If you do these right, performance reviews are forward-looking meetings

If both you and your direct are on the same page throughout the year and you've had them read your review a week in advance then why do you even need to meet? **To look ahead.**

Your meeting just became so much more effective because instead of lamenting on the past you can focus on the future. Now your performance review is about doubling-down on the things your directs have done well and weaving improvements into the areas your directs need improvement. The best way to do this is to show them the goals of your team or function and figure out **how your direct plays into achieving those goals**.

Now instead of your performance review being a recollection of the last year's work you can continue to charge ahead with _how last year can indicate what the upcoming year will be like_.

For this review season, for example, I spent about 10 of the 30 minutes pulling up the review, glossing over it with my directs, and leaving room for questions or concerns (of which there were exactly 0). For half of my directs, most of those 10 minutes were actually explaining how the performance review system works because this is their first annual review. The rest of that time was also spent explaining how our performance review system is going to change in this new year.

The remaining 20 minutes were focused on our function's goals for 2021 and how we can weave them into their teams' goals. We talked about the big things we want to accomplish and why they are perfectly positioned to take our business to the next level.

It was aspirational. It was inspirational. It was...exciting. Definitely not what you would expect to hear when you think of performance reviews but if you are doing all of the right things then it doesn't have to be painful; even if you have under-performers. I'd actually say this system is better for under-performers because they aren't caught off guard and they can instead be excited to improve in the future rather than be reminded of the past.
]]></content:encoded>
      <dc:date>2021-02-15T21:54:00+00:00</dc:date>
    </item>
    <item>
      <title>How I applied the engineering management curriculum</title>
      <link>https://www.adamconrad.dev/blog/how-i-applied-the-engineering-management-curriculum/</link>
      <description><![CDATA[Last week I posted a reading list of the books I have read (or plan to read this year) on engineering management. My concern is that no one teaches you this stuff in an MBA or at work so you’re kind of forced to do this on your own.

]]></description>
      <pubDate>Fri, 05 Feb 2021 22:28:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/how-i-applied-the-engineering-management-curriculum/</guid>
      <content:encoded><![CDATA[Last week I posted a [reading list](/blog/teach-yourself-engineering-management/) of the books I have read (or plan to read this year) on engineering management. My concern is that no one teaches you this stuff in an MBA or at work so you're kind of forced to do this on your own.

Further, it's a challenge because engineering management is like a mid-career switch. In many ways, you're starting over from scratch and many of the techniques you've applied as a software engineer no longer apply as an engineering manager. So where did I begin?

## I first explored some more universal topics

I have to be honest, I don't actually remember when I started reading from this reading list. At the time I was averaging nearly a book a week so it's difficult to pin down in the early days what I read first and in what order. But just reflecting back on the list I'm pretty sure I started with some more universally-applicable books like _How to Win Friends and Influence People_ and _Never Split the Difference_.

I read these two several years ago even before I was an engineering manager. I'm pretty sure _How to Win Friends_ was just a general recommendation on networking and making connections in the industry which has always been a concern of mine. _Never Split the Difference_ became relevant when I started my [consulting business](https://anonconsulting.com) and I wanted to make sure I was coming to the table prepared when my clients wanted to talk pricing.

Throughout the years I've built side projects I've explored lots of books around entrepreneurship so I think that's where _Traction_ and _Venture Deals_ came in but I know for a fact that was also at the time I was plowing through books. I'm pretty sure I read _Venture Deals_ in one day on a bus ride back from NYC and within the next week, I continued that trend with _Zero to One_ and _The Hard Thing About Hard Things_.

None of these are management or leadership books but I think it's important to give you context that **I read books at the moment I was experiencing situations I wasn't prepared for and wanted an expert opinion for how to handle them**.

## Solving problems as they occur

So given that criteria for reading the first engineering management book I read were when I was a tech lead who was offered the chance to have direct reports informally. And I quickly ran into a tricky interpersonal situation.

That's when a coworker recommended me _The 27 Challenges Managers Face_. I had experienced an exact situation that was spelled out in one of the 27 challenges in that book. So my first pass at this book was to solve that exact problem.

My next pass at the book was a quick read of the other 26 issues. I will say I haven't actually read this book thoroughly cover-to-cover because as I mentioned in the last article, I don't think this kind of book benefits from a cover-to-cover read. I think it's great to read quickly to internalize the problems you might face. Later, keep it in your back pocket as a reference guide when you run into tricky situations with your directs.

## My first management job

Once I officially earned my first formal engineering management position, that was when I started diving into my real reading list. It was prompted by a few aptly titled Ask Hacker News posts about how to transition into Engineering Management.

There was one comment that recommended the _Manager Tools_ podcast and so I went right to their website and looked for the basics. That's where the "Management Trinity" was first introduced to me. That weekend I put these three podcast episodes on repeat. Any workout I was listening to them over and over again. I even think I listened to them as I was falling asleep, hoping I would retain this information subliminally.

The podcast is I think over a decade old so once I started listening to the other episodes I heard about the _Effective Manager_ book as a cross-promotion for the podcast so at some point I got the book and read that.

Again, if you've listened to the podcast you've already hit the main talking points. These were the things that taught me the engineering management basics: **1-1s, coaching, feedback, and delegation**. The book and the podcast drill those concepts into your brain with lots of examples.

So for at least the first 6 months of my new management job, all I practiced were those 4 concepts until I felt comfortable in all of them. And then 3 months later I quit.

## Starting my consulting firm

After my management job I read most of the chapters in _Cracking the Coding Interview_ because I figured I'd have to do a bunch of coding interview questions even as an engineering manager. I didn't find any companies I liked that were hiring engineering managers so I started my own consulting firm and became a freelance developer.

At this point, I really kicked up my reading because, in addition to managing teams as an outsider, I had to learn all about marketing, sales, and all of the non-technical things I have never learned before.

This is when I read books like _SPIN Selling_ and _Never Split the Difference_. You won't see most of these books in my recommended reading list because that's for a completely different curriculum on starting your own consulting business which I may someday create a curriculum for as well.

I only ran this company full time for about a year and a half; it would have been longer but my next client turned out to be my next full-time job.

## My current company and current role

I'm currently working at the company I thought was going to be my next consulting gig. I passed all of their hiring tests and everything. As soon as I was ready to sign on the dotted line with my consulting engagement they also offered me a full-time position in case I wanted to take that instead. The offer was better than I expected and it offered health insurance so I took the offer and went back to full-time.

For the first 6 months or so I was in an individual contributor role so my management training took a backseat. Soon enough, a management opportunity opened up, and, leveraging my previous management experience and training, I applied for the role of director (the job I currently have).

Once I landed director I knew I had to level up my learning yet again. I was back into voracious book reading mode.

Since I had nailed the basics with _The Effective Manager_ I didn't feel the need to re-read any older material so I dove right into books that were catered for more senior management. I was reading a bunch of blogs at the time on the topic so I started plucking books from their recommended reading lists. _High Output Management_ came up time and time again so I read it in a weekend.

As I stated in the last article, I didn't get too much out of it that applies to directors directly but I did cross-reference the 200-point checklist that is listed at the end of the book to ensure I covered as many aspects as I could with the team that I had.

I can't name any of the tactics off hand right now but I would just encourage you to read the book and nail that checklist. You'll see that unless you're a VP or CTO you're going to have a hard time being able to act on all of the things Grove recommends.

This is more true today because he ran a hardware company in the 90s so doing things like "walking the floor" on any of the assembly lines simply won't apply to the folks that read this since you're most likely managing software teams.

The other books I read around this time were _The Talent Code_ and _The Little Book of Talent_, both of which were recommended in the Drift company podcast time and time again. These were the books where I applied **identifying top performers** in both retention and hiring. It's also a great way to identify talent habits for yourself in improving your own performance at work.

### Director-level learnings

The first book that really clicked for me as a director was _Extreme Ownership_. Alongside books, I also listen to a lot of podcasts. Jocko Willink's podcast is a favorite of mine and just like the _Manager Tools_ podcast, I heard about the corresponding book enough times that I had to read it.

This was the first new concept I learned in a while: **ownership and accountability**. I knew about these things, sure, but this was the first time I took the ownership part to a new, _extreme_ level. I know, cue eye roll.

But unlike in the past, this was the first time I felt the gravity of my role as a director. I wasn't just making decisions for one team, I was making decisions for lots of teams and lots of people, even some that I didn't directly interface with.

I've heard this saying before: _the further up the career ladder you go, the longer and harder it is to see your successes or failures_. That means the initiatives you drive today might take months or years to see any concrete results or changes in culture.

That's crazy timing because just today I received upward feedback from a skip-level report that I was praised for having been one step ahead of the company on certain initiatives such as building out OKRs that channel down from company or organizational goals.

The truth is I had no plan to be ahead of the curve, I simply saw what other companies were doing, experimented with some tools and tactics, and doubled down on the effective ones. OKRs seemed like a no-brainer to me and I started implementing them over a year ago. More than a year later I am only just receiving feedback from my directs that this was a good tool to use for our teams.

This relates back to the _Extreme Ownership_ because I know everything I teach my teams comes back to me. Every success and every failure I have to own.

It's really easy to say "this person was late and it is their fault for being lazy." It's really hard to see within yourself that they might be late because you never told your teams that arriving promptly for meetings is a cultural habit you want to build.

It's also really easy to say "this person missed the deadline because they didn't work hard enough." It's really hard to see within yourself that they might have missed that deadline because you didn't check in with them when they got stuck and that you didn't reinforce why the deadline was so important to hit in the first place.

At the end of the day, **it _all_ comes back to you**. I'm glad I learned this lesson now.

### A military theme

The next book on my list came from a recommendation from yet another Ask Hacker News thread...because we, as engineers, have never seemed to figure out the transition to engineering management.

I just so happened to have finished a book from a military officer so it was quite impeccable timing that one of the top comments on this thread was about a free leadership book produced by the US Army. I figured if they can train millions of Americans to fight in the most powerful army in the world then they probably have figured out how to train junior officers and generals alike.

_ADP 6-22_, which sounds like the name of a Star Wars robot assassin, was a very short and to-the-point read that I got through in one night. I should probably read this one again. This is like _How to Win Friends and Influence People_ in that it is so fundamental that you learn something new every time you go back and read it. This is more than just a book for leaders like me, this is a book for every kind of leader in every kind of profession.

Because of this broad appeal, I am saddened to say I can't recall any lessons that I haven't already learned elsewhere. I mean we're talking real basic stuff like how to communicate effectively and how to report up the chain of command. These are all such cornerstone topics I couldn't tell you if I learned them because of this book or if I learned them years ago and they were resurfaced through this book.

This post is a reminder to myself that I have to read this book again. It's also a reminder that when I finished reading this book I recommended Jocko Willink cover it on his podcast. He replied back to my tweet but I'm pretty sure he hasn't covered it yet.

In looking for that podcast episode I just discovered the Army [also published an audiobook of this manual](https://rdl.train.army.mil/catalog-ws/view/ADP6-22-Audiobook/index.html) so if you'd rather listen to the book now you have 0 excuses. Both are free to download from the US Army website.

### High octane learning

The last three books I've read on engineering management were probably the most rewarding and the most tactical since I had listened to the "Management Trinity" over three years ago. These were _Accelerate_, _The Manager's Path_, and _An Elegant Puzzle_.

_Accelerate_ teaches you about the [DORA metrics](https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance) which completely revolutionized how I thought about measuring the performance and effectiveness of my teams. It radically simplified my quest for understanding how strong my developers were without having to resort to controversial tools like Code Climate Velocity and Jellyfish. This book on its own can transform how much work your teams get done through their DevOps practices.

_The Manager's Path_ does a great job of spelling out the typical career ladder for a manager. Since I didn't read this book until I was a director, I mostly skipped over the previous roles I had already attained: tech lead and front-line engineering manager.

Director is the next major leap because you go from leading individuals to leading teams via other engineering managers. Now you're not coaching individuals on how to write effective code, you're teaching managers all of the things you were supposed to have learned yourself in the books I had already read.

This book helped me dial in on a few new concepts I hadn't really practiced before: **skip level 1-1s, coaching managers, and peer management**.

As an aside, I can't imagine being a director without having read so many books beforehand. What is it like to teach other managers if we're already really bad at teaching our craft?

I guess this can come through hard-fought experience as well which is why I can't stress enough that **books are one of the cheapest, highest ROI ways to accelerate your career**. There is no way I would be successful as a director today if I didn't make up for the lack of job experience with a massive amount of self-education.

_An Elegant Puzzle_ is the last book I've read on engineering management. It's a pretty large book and I've been slowly making my way through it since the past summer. A big reason why it has taken so long is that I learn something new on nearly every page. I find myself reading and re-reading pages because the nuggets are so golden I have to stop and take a second look.

From sizing engineering teams to delivering presentations to senior leadership, these are the kinds of duties I have to perform as a director. I think one of the reasons I've been able to stay successful in my current role is because of this book. There are literally too many lessons I've learned from this book that it would take me an entire series of blog posts to synthesize them all so I would just encourage you to read the book for yourself.

## What's next

The only two books that were in the curriculum that I haven't mentioned yet are _Nonviolent Communication_ and _The Trillion-Dollar Coach_.

I read _NVC_ a year ago right before we kicked off a big feedback week as a company but I found I had already internalized this framework from the feedback topic on the _Manager Tools_ Management Trinity. The exact, step-by-step framework that Horstman recommends for delivering feedback is almost entirely based on _Nonviolent Communication_. If you've read _The Effective Manager_ you don't need to read this book if you specifically want to deliver feedback as a manager.

I still haven't read _The Trillion-Dollar Coach_ because as I stated before I think the lessons here are likely above my pay grade. That said, I'm still probably going to read it next because I'm really intrigued to read about how this guy has managed to persuade so many top executives to utilize him as a coach. That story alone is astounding enough that I want to read about it and maybe I'll pick up so useful management topics.

Finally, if I were to organize the upcoming reading list from my previous post of 40 books, I'd say the next three after _The Trillion-Dollar Coach_ would be: _Resilient Management_, _Turn the Ship Around!_, and _The Five Dysfunctions of a Team_. I'm incredibly intrigued by _Resilient Management_ because I respect Lara Hogan's work and read her blog. I've heard _Turn the Ship Around!_ more times than I can recall and I am a sucker for any book that has a military theme wrapping management concepts. _The Five Dysfunctions of a Team_ is yet another book that has been recommended so many times in my circles that I have to read it sooner rather than later.

I'm sure there are plenty of books I should have read earlier in my remaining list of 40 books and as I read them I'll be sure to update my curriculum but I'd also love to hear from you what books you've found most valuable in your education as an engineering manager.
]]></content:encoded>
      <dc:date>2021-02-05T22:28:00+00:00</dc:date>
    </item>
    <item>
      <title>Teach yourself engineering management</title>
      <link>https://www.adamconrad.dev/blog/teach-yourself-engineering-management/</link>
      <description><![CDATA[Engineering management is rarely taught to new managers.

]]></description>
      <pubDate>Sun, 31 Jan 2021 21:09:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/teach-yourself-engineering-management/</guid>
      <content:encoded><![CDATA[Engineering management is rarely taught to new managers.

The usual path to management looks something like this: your manager identifies you as someone with potential but there are either (A) no more ranks to climb as an engineer or (B) no good managers they can hire so they put you in the manager slot.

This is troublesome for two reasons:

1. Not everyone wants to be a manager
2. For those that do, they're thrust into the job without any training or help

If you're one of the lucky managers who actually wanted to make this change then you _need_ to learn how to manage effectively.

This curriculum is based on what I taught myself over the last 4 years to climb the ranks from engineering manager to director. This curriculum is also the same one I will use to continue up the ranks to VP and CTO as I believe the advice and needs change as you move higher in an organization.

I'm going to arrange this much like you would a computer science curriculum: start with some core courses (books) and then bridge into more advanced concepts, with optional side tracks for certain specialties (or weaknesses) to focus on as you gain more knowledge.

## Freshmen Core Curriculum (Mandatory Reading)

I consider this to be the base of all learning for engineering managers - from line managers who call themselves "Engineering Manager" all the way up to founders. Anyone can get something great out of these books and they are the ones that are recommended the most often:

### The Effective Manager & The Manager Tools podcast

When I first started out I just wanted someone to walk me through the basics. That's exactly what this book and the podcast do for you. I'll simplify it even further: [listen to the Manager Tools Trinity](https://www.manager-tools.com/2009/10/management-trinity-part-1). This is your 101 intro course.

**If you took nothing else away from this post, listen to the Trinity.**

It is opinionated. It might even be considered controversial. But if you nailed **one-on-ones, feedback, coaching, and delegation** you would be 80% ahead of your peers.

The book expands on these 4 concepts with examples aplenty but I'd recommend you listen to these 3 podcast episodes on repeat until every concept from each episode is drilled into your brain. Nothing has carried me as far into my management career as these three episodes.

### Extreme Ownership

This one is so foundational that I would argue this is for everyone, not just managers. In a way, I could consider this to be the prologue to the management curriculum.

This book was written by a Navy SEAL officer who says that **you own everything in your life.** And I do mean _everything_. Every failure, every win, and every stalemate is attributed to something you have done.

Your developer didn't push code with tests? You own that because you didn't ensure they internalized the policy on tests must be delivered with code.

Your team missed a deadline? You didn't reduce the scope and have an exit strategy for what could happen if the milestones slip.

No matter what the question, it all comes back to _you_. The magical thing is once you internalize this concept and lead by example, _everyone_ on your teams picks up on this and they also share in that ownership. It's an accountability feedback loop that only trends in a positive direction.

### Army Leadership and the Profession (ADP 6-22)

Speaking of military books, here's another leadership book is taken directly from the US Army that you can read for free [here](https://armypubs.army.mil/epubs/DR_pubs/DR_a/pdf/web/ARN20039_ADP%206-22%20C1%20FINAL%20WEB.pdf).

I didn't find this book until I was a director but this is one of the highest-recommended books on leadership I've ever come across. _This_ is an actual Leadership 101 book. It covers everything from mindset to effective communication to how to actually lead and get others to follow.

While _TEM_ is the most tactical 101 book for your needs as an engineering manager, this is the true 101 must-read for any manager in any profession. If the world's most powerful military uses it to train their officers then I would imagine it can be a great foundational read for you as well.

### The Manager's Path

Once you've nailed the tactical basics with the right mindset, only then can you begin to see the hierarchy of how one develops into a manager from a technical lead. This is a bit more high-level than the other two books because it covers the whole path from tech lead all the way up to VP and CTO.

I like this book because it helps you understand the relevant concepts that are most important at each level. I reference this book every once and a while even as a director just so I can refresh my memory on what is most important at my current level and how to start thinking about operating at the next level.

### The 27 Challenges Managers Face

This is your dictionary and thesaurus of engineering management. As the book says, it walks you through how to handle 27 of the most common challenges with the people on your teams. Everything from under-productive developers to personal conflicts, this book walks through each scenario step-by-step.

Because you're never going to memorize, internalize, or encounter all 27 issues when you first start out, this book is best left to skim once to keep in the back of your head so that when you do encounter a situation, you can remember it was covered in this book and to quickly look it up for actionable guidance in your difficult situation.

## Sophomore Curriculum (for Directors)

Honestly, you could stop here and you'd be way ahead of other managers in the field. At some point, you'll move up in your career and you'll need to focus on a different set of things.

Instead of learning how to hire engineers and conduct interviews, you're learning how to create hiring programs. Instead of hiring for a budget, you're drafting the budget. Instead of building a team, you're building many teams and training those managers to run those teams autonomously.

These are the books for your next set of challenges:

### An Elegant Puzzle

This is arguably my favorite engineering management book. Like _The Effective Manager_, it has play-by-play guidance on extremely important topics to managers. The difference is that _TEM_ covers the tactics to manage individual contributors (ICs). _An Elegant Puzzle_, however, covers the tactics to manage and build teams.

As a director, this is my CLRS _Algorithms_ book. It is both my bible and my reference guide for all of the major facets of my job as a director. It covers everything from how to size engineering teams to how to build a hiring funnel.

Given that Will wrote this as a tech leader at Stripe (and is now a CTO at Calm) this book is useful for anyone at the director stage and above. I am sure I will continue to reference this as I transition into senior management roles.

### The Talent Code

This is about how to nurture talent and bring out top performers. I debated whether or not to add this to the Core Curriculum because it is something any manager would benefit from learning more about. However, this is more of an advanced concept in coaching and arguably about identifying top talent as a hiring manager.

Therefore, I just don't see this as a core reading but if you are so inclined to dive more into the coaching aspect of management, this is a great book.

There is also _The Little Book of Talent_ which is like an abridged version of this book with 52 specific tips and traits for fostering talent which is an acceptable substitute for this one if you didn't enjoy this book. Personally, I like _TLBOT_ more since I found the backstories of some of these talent aspects to be ranting diatribes but to each their own.

### Accelerate

This book is not about management. But once you become a director you'll want to start measuring the performance of your teams to help identify bottlenecks, weak points, and performance gaps. _Accelerate_ provides the most robust, scientifically-backed means with which to do so.

The book is the foundation for the [DORA metrics](https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance) (also known as the "Four Keys") so you could theoretically skip the whole book and read just this blog post and you would get the gist of what they are trying to accomplish.

If you do want to read more, the actual meat of the study and the takeaways are in the first half of the book. If you're a real nerd for the scientific method, the second half of the book goes into the details of how the study was conducted and really dives into _how_ they determined the results while the first half focuses on _what_ the results were.

## Junior Curriculum (for VPs)

As I mentioned before - I've never been a VP of Engineering so I can't speak to whether these are good books for VPs, but they are books I have read and plan to revisit when I get there because I found them to be unnecessary for my journey so far.

### High Output Management

To start, this is a great book and is in the engineering management canon for a reason. That reason is it was written by the former CEO of Intel who helped transform the company into the leading semiconductor outfit in the world.

But that's exactly it - it was written by a CEO. The things he needs to think about to run a successful organization are not what you need to worry about as a line manager or even as a director. All of the other books above should come before you read this book. I know this because I read it and I don't remember what I retained from it.

I know it was a great read at the time but the only thing I can remember that applies to my work is that there was a massive 200-something point checklist at the end of the book to evaluate yourself and your organization in areas for improvement. This is exactly the kind of list that is useful for someone who runs an entire engineering organization but is major overkill if you don't.

And even then, this book was written in 1995 so there are a lot of stories and concepts that are both dated and more relevant to the hardware industry (while I primarily cater to software leaders).

I'll definitely reference this for the next company I join but if you read the other books first you'd be in a great spot.

### Trillion Dollar Coach

Warning: I haven't read this book yet. That alone should be a reason why you can understand why it is in the higher-level curriculum than my pay grade. Why?

So far, everyone who has recommended this book to me has either been a CEO or a CTO of a startup. And this book is about engineering startup playbooks. This tells me that it's all about building engineering teams with top-brass leaders in mind.

The title gives that away: the term "trillion-dollar coach" comes from the fact that this management consultant has helped guide the chief executives at Apple, Google, and Intuit, who have a combined market cap well into the trillions.

Browsing the table of contents, terms like "team first" and "envelope of trust" are the kind of visionary, aspirational wordsmithery of those who command a very senior title. Therefore, this is a book I want to get to but I'd rather work through much of my reading list first before I get here.

## Senior Curriculum (for CTOs and Founders)

At this level, you're writing checks or you're pitching to the people who write those checks. If line managers are tactical, directors are strategic, and VPs are visionary, then you are the foundation and the exemplar. Everything stems from you.

You might be a technical co-founder or you're incoming CTO of the entire United States in which case thanks for reading my blog, I have no idea how you got here or why you're taking advice from me.

I have co-founded [a venture of mine](https://gettoknowapp.com/) which is just me and a friend so I can't say these books help in managing humans but they have been useful as the CTO of that outfit.

### Venture Deals

This is a great founder 101 book because it explains all the nuances of setting up a software company, scaling it, and all of the nuances about raising capital and pitching to investors.

While it is definitely framed more for founders than for leaders at established companies, there is some _excellent_ wisdom in here about how to organize your company. There is a great exercise in here about how you split up all of the executive roles based on the number of founders you have and how to complement skill sets based on all of the functions a company needs to operate, such as marketing, sales, and engineering.

What resonates with me as a leader is that **building teams require some combination of redundancy and complement in skill sets.** Know what skills you need to bolster (e.g. front end) and what skills are lacking to round out your teams (e.g. platform engineering) and it will be easier to scale with the needs of your organization.

### The Hard Thing About Hard Things

I'll admit, this is one of those pop non-fiction books that have a lot of cool stories marrying Ben Horowitz's love of technology and hip-hop. I came for the stories and stayed for a few nuggets of sage advice. The biggest one being **nip rumor culture in the bud.**

This one I noticed from a previous startup and when it's bad, it is toxic. Knowing what it looks like first-hand and how to resolve it (thanks to this book), I'm much more aware of how to maintain a strong culture with integrity and honesty now and how information needs to travel from the top down.

The other one I remember is to **take care of your people.** Empathy, trust, and psychological safety are [tops on my list](/blog/technical-lead-management/) of skills to build as a leader and there are a lot of examples where Ben was put in a rough situation and had to either fire someone compassionately or ask way too much from his employees. He knew the only way he could ask so much as if he spent the rest of his time caring for his people.

That kind of caring takes _years_ to build with your teams and the only way you will get people to join your crazy ride is to deeply care about how they are as people.

### Zero to One

This is another pop non-fiction book from Peter Thiel that everyone seems to reference but is really full of stories and lacking in major tactics. You walk away from books like this and _THTAHT_ with like one or two nuggets and appreciate for what it is: a fun read feigned as education.

Thiel really lets his libertarian pride shine in this one. I'm not trying to get political here but the main takeaway I got from this book is that **moats are powerful** and **crush your enemies with a monopoly.**

I think if I were running a venture-backed firm with a bit more fire under my ass this book would have spoken to me more but I found it a bit too cutthroat for me. Your mileage may vary and it is a highly-recommended book so I'm still willing to include it here.

### Traction

Another book that is not about management. But the only way you'll grow big enough to be a manager of people is to build something people want so much that you need to hire folks to build it fast enough.

_Traction_ has a ton of great insights I've used to build my side projects into profitable ventures with tactics on how to get users. I haven't even begun to use all of the tricks in this book but it is absolutely a worthwhile read particularly if you are an early-stage software startup founder.

## Specialty tracks and additional books I've enjoyed

Many college CS programs offer you the opportunity to specialize in certain disciplines within computer science after you learn the basics. These include artificial intelligence, programming languages, and human-computer interaction.

Similarly, engineering management has a few specialties as well and I've read a few books that delve into very particular facets of the management craft:

### Influence, persuasion, and soft skills

_How to Win Friends and Influence People_ is a classic among the classics. I think it is one of the most highly-read non-fiction books of all time.

The title is so to the point you might hide the cover on the subway for fear you might look like a social Luddite. It is not that kind of book. This book is not how to make friends for dummies.

This is a book on getting others to do things you want _without losing your moral compass_. Influence is a very neutral term: to some, it sounds like something slimy a salesman does to convince you to sign on the dotted line. To others, it sounds like the kind of charismatic tactic President Clinton used to win over voters in the presidential debates.

I'd say this book is one of those gentle reminders that **you should be nice to people and that being nice to people makes it much easier to get them to see your side.**

This book just has to be read (and re-read) to be believed. It should be on everyone's reading list for the simple fact it's so widely accessible and in every library that it's basically free.

### Health and productivity

_How Not to Die_ is a mammoth text at over 570 pages. It's not a management book. It's not a software book. It's not even a self-help book by how you would commonly define that genre of book. This is a book on healthy eating.

In a year like we've just had, health has become a front-and-center reality for us all. Healthy habits have [been shown](https://www.cdc.gov/workplacehealthpromotion/model/control-costs/benefits/productivity.html) to increase productivity.

I live by six healthy habits that doctors recommend every day:

1. Eat right
2. Exercise
3. Don't smoke
4. Drink in moderation
5. Reduce stress
6. Get enough sleep

This book tackles habit number 1. It explains how [this doctor's list of a dozen food groupings](https://nutritionfacts.org/video/dr-gregers-daily-dozen-checklist/) help reduce all-cause mortality for all but one of the top 20 reasons people die (the one it can't help are car accidents). From obesity to cardiovascular disease, all of these foods and habits help you improve your health through the basics we've learned all along: a diet rich in whole fruits, vegetables, legumes, and grains is shown to improve health.

There is even an app for both [iOS](https://apps.apple.com/us/app/dr-gregers-daily-dozen/id1060700802) and [Android](https://play.google.com/store/apps/details?id=org.nutritionfacts.dailydozen&hl=en&gl=US) to help you track your daily progress on eating these key foods. I use it and my Apple Watch as one of my key health staples for ensuring I meet these 6 habits every day.

A healthy manager is a productive manager operating a peak performance.

### Giving feedback

_Nonviolent Communication_ spells out a four-step process for delivering feedback. This one was a game-changer for me on how to speak to people in a way to promote behavior change without casting blame or making people feel bad.

If you want the ultimate deep-dive into how to deliver feedback that resonates with your directs, this is a book to read and study carefully.

### Salary negotiation, hiring, and retention

_Never Split the Difference_ is great whether you are looking for a new job or you are trying to hire a key candidate. It was written by an FBI hostage negotiator who takes a pretty hard stance on how to jump into a negotiation.

This is another one of those books that take 300 pages to convey a few key points as most pop self-help books do these days. I thought the book was entertaining so I didn't mind the fluff but the biggest takeaway I got from this book is the phrase **how am I supposed to do that?**

If I recall correctly, the author was put on the spot to negotiate a really tough situation like how to give terrorists millions of dollars or else they will kill the ambassador. Most people who read this panic and have no idea what to do and figure the author is totally trapped into this nearly-impossible scenario. His response is clutch: "how am I supposed to do that?"

In other words, **if the person on the other side of the table is being unreasonable, turn their poor offer back on them through empathy**. Let's say you're making a cool $150K at your current job and the company interviewing you just offered you $110K for acing their interview.

You can reply with "how am I going to be able to take that offer when I'm already making more than that?" You're putting the onus on the hiring manager to come up with a really good reason why they should make the obviously irrational choice of choosing to leave their current job for a worse one with a lower salary. This forces the hiring manager into a situation where they have to come back to the table with a stronger offer.

As a manager, you can use this tactic as well...but I've already said enough about this book. It's great and it goes into a bunch more tactics on negotiation which you can use when looking to hire new members for your team(s).

### Interview process & design

_Cracking the Coding Interview_ is the canonical reference for developers looking to pass the interview at the FAANG companies and land that big salary. The whole purpose of this book is to game the system like one does for studying the MCAT or SATs with practice questions used at real companies.

**I'm _not_ recommending this book so you can just copy questions from here to use in your interviews.** In fact, I am recommending the opposite: this book is a great way to identify which questions are complete _nonsense_.

If you're a web startup you have no business asking JavaScript developers about bit manipulation and bit shifting. Is it useful knowledge? Possibly. Do any of you read this article actually use heavy bit shifting techniques in your code? Extremely unlikely.

The only time I found a bit manipulation interview question praiseworthy was an interview I did many years ago for a hardware company that built IoT devices. They actually used these techniques and the interview question was actually really fun.

For the rest of you, this is a net-negative and just makes you look pretentious and leading to more false negatives than you can stand to lose in the hiring process.

The bonus for reading this book is if you ever decide to become an IC again you'll have no trouble getting a job or grinding LeetCode.

## Books on my list that I haven't organized yet

Finally, here are the books on my reading list as a manager that I haven't read before and I don't know where to bucket them in the curriculum yet. As I read them I'll update this post with where they should fit. For now, here are the books on my reading list  that I hope to get through in 2021:

* _The Art of Leadership_
* _The Charisma Myth_
* _The Coaching Habit_
* _Confessions of an Advertising Man_
* _Content Inc_
* _Deep Work_
* _Difficult Conversations_
* _The Effective Executive_
* _First, Break All the Rules_
* _The Five Dysfunctions of a Team_
* _High Growth Handbook_
* _It's Not How Good You Are, It's How Good You Want To Be_
* _Leadership Step by Step_
* _The Making of a Manager_
* _Managing Humans_
* _Managing the Unmanageable_
* _Measure What Matters_
* _Mindset: The New Psychology of Success_
* _No Hard Feelings_
* _The ONE Thing_
* _Only the Paranoid Survive_
* _The Phoenix Project_
* _Quiet Leadership_
* _Radical Acceptance_
* _Radical Candor_
* _Resilient Management_
* _Righting Software_
* _Running Lean_
* _Software Engineering at Google_
* _Sprint_
* _Start with Why_
* _Strategic Logic_
* _Team of Teams_
* _Team Topologies_
* _Technical Leadership and the Balance with Agility_
* _Technology Strategy Patterns_
* _Thinking in Promises_
* _The Timeless Way of Building_
* _Turn the Ship Around!_
* _The Win Without Pitching Manifesto_
* _Your Code as a Crime Scene_
]]></content:encoded>
      <dc:date>2021-01-31T21:09:00+00:00</dc:date>
    </item>
    <item>
      <title>Start today</title>
      <link>https://www.adamconrad.dev/blog/start-today/</link>
      <description><![CDATA[I don’t like when people ask me what my New Year’s Resolutions are.

]]></description>
      <pubDate>Thu, 31 Dec 2020 21:30:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/start-today/</guid>
      <content:encoded><![CDATA[I don't like when people ask me what my New Year's Resolutions are.

I don't have any and I never will.

If I want to resolve to do something - I just do it. Right then and there. I'm writing this post on the night of the last night of the year not because it was meant to close out one of the worst years in the history of my life but because there is no reason to wait.

Let me repeat that: **there is no reason to wait**.

January 1st doesn't usher in some reset button like how turning 26 magically drops your car insurance rates. January 1st doesn't absolve you of the things you missed yesterday that you know you could have physically, emotionally, or spiritually started yesterday. January 1st doesn't care.

**Resolve to never make arbitrary resolutions again.**

Resolve to take charge of your life and the things you want to accomplish _the moment you want to accomplish them_. Inertia is a powerful thing and that first push is the easiest when you are most motivated to start: **the moment you consider it a possibility**. As soon as you know you can (or want) to achieve something is the time when you have the motivation and inner encouragement to make that a reality.

2021 will be better. But you don't have to wait for tomorrow to start living better today.
]]></content:encoded>
      <dc:date>2020-12-31T21:30:00+00:00</dc:date>
    </item>
    <item>
      <title>A quick and dirty guide to Kanban</title>
      <link>https://www.adamconrad.dev/blog/quick-and-dirty-guide-to-kanban/</link>
      <description><![CDATA[I recently saw a question about how to implement Kanban and I realized I could very succinctly describe what is involved in the practice. Lots of people write whole books and courses on Kanban but I think that is completely unnecessary. I propose that you can learn (and implement Kanban) in about 5 minutes with a quick, one-page read.

]]></description>
      <pubDate>Mon, 30 Nov 2020 16:18:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/quick-and-dirty-guide-to-kanban/</guid>
      <content:encoded><![CDATA[I recently saw a question about how to implement Kanban and I realized I could very succinctly describe what is involved in the practice. Lots of people write whole books and courses on Kanban but I think that is completely unnecessary. I propose that you can learn (and implement Kanban) in about 5 minutes with a quick, one-page read.

This guide is purposefully short because I want you to know that **there really isn't all that much to this practice** so I'm going to save you all of the fluff and background that usually accompanies one of my posts.

**Kanban is just a way of organizing your teams' work.** It's all about _continuous improvement_ and the _acknowledgement that priorities change constantly_.

I think **most software teams that don't have hard deadlines would benefit more from Kanban than Scrum**. The reason is that software is inherently complex and often has moving targets. So gaming companies, which do have hard deadlines for when their games are released, probably won't benefit from Kanban and should stick to the Scrum (i.e. waterfall) approach. If you're a SaaS software product building things on-the-fly, consider Kanban. So what is involved?

## 1. Put your tasks on a visual board

So many tools exist that already do this: Trello, Asana, and Jira are three I can think of that already do this aspect of Kanban: **become aware of what you are _actually_ working on**. Just showing everything to your teams helps you understand where the blockers, bottlenecks, and underutilized capacity lies.

## 2. Create a simple workflow to start

I would recommend a 5-step workflow that has scaled to multiple teams on very large products:

1. **Inbox:** The stuff that just came into your backlog that is yet to be prioritized
2. **In scope:** The stuff that you've prioritized and could actively pick up to work on
3. **In progress:** The stuff someone is building with code and tests
4. **In review:** The stuff that has a Pull Request or some other kind of gate to review quality of the work to be delivered
5. **Done:** The stuff that is out on your production servers and released to customers

## 3. Use the workflow and check for blockers

With this workflow you'll see tickets moving from one step to the next. The number of tickets in each column tells you something. When tickets pile up in a given column here is how I investigate them:

1. **Lots in inbox:** Check with product management to find time to prioritize - either lots of new ideas are being sourced (good) or lots of bugs, issues, or debt are accumulating (bad).
2. **Lots in scope:** The sprint just started (not an issue). If the sprint has been going on for a while, either the team committed to too much, too many bugs came into the sprint, or there is a bottleneck further up the workflow.
3. **Lots in progress:** _You should only ever have as many tickets in progress as there are people_. If there are fewer then not everyone is working on something and if there are more then someone is juggling more than one task at a time. **The whole thing with Kanban is that a great workflow means doing one thing to completion.** So don't let anyone work on more than one thing at once (in terms of active development - you could have a ticket in review waiting for PR feedback while you are starting on another ticket)
4. **Lots in review:** Developers are not paying attention to Pull Requests and product/design are not checking the acceptance criteria are being fulfilled. My teams provide a guarantee of 4 hours to review their teammates' code. Make sure you have [code owners set up](https://help.github.com/en/github/creating-cloning-and-archiving-repositories/about-code-owners) and ensure all of the right parties are seeing notifications that they need to review the code.
5. **Lots in done:** Nothing wrong with that.

## 4. Refine your workflow

Start with this template and see what works for your teams. Maybe you have a staging server that needs to be pushed for review before you can push to production so insert a column between steps 4 and 5. Maybe you have separate, distinct steps for Code Review, Design Review, and Product Review - add them in. **What matters is that you take the basics of a Kanban flow and iterate on your workflow to suit the productivity of your teams.** And further, each team may have a different workflow that works for them so **be open to adopting not just one workflow but many**.

## 5. Measure and learn

Okay so you did all of this stuff and now you're running Kanban (all of this can be turned on with a switch [in Trello](https://trello.com/b/C4Awm5lK/kanban-board) or [Jira](https://www.atlassian.com/agile/tutorials/how-to-do-kanban-with-jira-software) or [Asana](https://asana.com/uses/kanban-boards)). How do you know this was the right move? Kanban is great for measuring a few things which correlate really well with [high-velocity teams](https://www.amazon.com/Accelerate-Software-Performing-Technology-Organizations/dp/1942788339?SubscriptionId=AKIAILSHYYTFIVPWUY6Q&tag=duckduckgo-ffab-20&linkCode=xm2&camp=2025&creative=165953&creativeASIN=1942788339):

1. **Cycle time**: How long does it take for completed dev work to actually get into the hands of customers? High-performing teams have a very short feedback cycle for review and release.
2. **Lead time**: How long does it take for a ticket to arrive on the door of the backlog and get it all the way to the done step (i.e. how long does it take for someone to ask a developer to do something and then have it delievered to customers). Cycle time is a subset of lead time but is important to distinguish to help bring to light the things that are in the control of the developer (planning, prioritizing, and working on the product) and out of the control of the developer (the review and release process). Just like cycle time, lead time should be fast and fluid.
3. **Throughput**: How much stuff gets put into `DONE` in a given sprint? Is it trending up or down or stagnating? High-performing teams can get through a lot of stuff quickly (duh).
4. **WIP** (work in progress): Are people actually focusing or trying to juggle a lot? High-performing teams have a low WIP (ideally 1 per developer).

If you can measure these 4 things and trend them in the right direction, this should indicate that it was a good decision to adopt Kanban.
]]></content:encoded>
      <dc:date>2020-11-30T16:18:00+00:00</dc:date>
    </item>
    <item>
      <title>Don't call it tech debt</title>
      <link>https://www.adamconrad.dev/blog/dont-call-it-tech-debt/</link>
      <description><![CDATA[I think we, as a profession, shot ourselves in the foot when we coined the term tech debt. This recent post (though my browser’s search bar found no less than 3 recent articles I read on the subject in Hacker News) got me thinking that the term tech debt is an exercise in bad marketing.

]]></description>
      <pubDate>Mon, 16 Nov 2020 23:31:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/dont-call-it-tech-debt/</guid>
      <content:encoded><![CDATA[I think we, as a profession, shot ourselves in the foot when we coined the term _tech debt_. This [recent post](https://daverupert.com/2020/11/technical-debt-as-a-lack-of-understanding/) (though my browser's search bar found no less than 3 recent articles I read on the subject in Hacker News) got me thinking that the term tech debt is an exercise in bad marketing.

Debt is evil. It's the stuff that can [live on](https://money.com/student-loan-forgiveness-death-discharge/) longer than you do. There's never been an instance where calling something "debt" elicited a positive response. So why do we tell our Product Managers we have to _tackle_ tech debt?

Is it to create urgency? I hope not. Everyone believes their priorities are the most important priorities. So an engineer's tech debt will never seem more important than a PM's product backlog.

Is it to create gravity around the seriousness of the work involved? Then why is it that PMs always bucket "20% time" for making room for tech debt that is never fully paid off?

I loved this analogy I saw on Hacker News about this concept:

> If you run a commercial kitchen and you only ever cook food, because selling cooked food is your business -- if you never clean the dishes, never scrape the grill, never organize the freezer -- the health inspector will shut your shit down pretty quickly.
>
> Software, on the other hand, doesn't have health inspectors. It has kitchen staff who become more alarmed over time at the state of the kitchen they're working in every day, and if nothing is done about it, there will come a point where the kitchen starts failing to produce edible meals.
>
> Generally, you can either convince decision makers that cleaning the kitchen is more profitable in the long run or you can dust off your resume and get out before it burns down.

On the flip side, if you ever have the luxury of getting ahead of your technical debt then you can contribute things called [technical credits](https://www.stillbreathing.co.uk/2016/10/13/technical-credit). As you can imagine, technical credits are the opposite of tech debt: things that _pay dividends_ to your code rather than incur a cost and drag it down. [Progressive enhancement](https://gomakethings.com/progressive-enhancement-graceful-degradation-and-asynchronously-loading-css/) is touted as an example of technical credit.

Marrying these two concepts, I actually think a way around the horrible marketing of tech debt can be resolved with Agile terminology that is already familiar to our product-positive counterparts: **engineering-driving work.**

## Engineering-driven work over tech debt

At the end of the day, whether they be debt or credit, the work engineers want to accomplish to make their code better is something personal to them and them alone. Similarly, your product team has their own vision of what your code can become for your customers. So if we segment on the _owner_ instead of the _outcome_ we remove the negative connotations of the work involved.

On my teams, we make this distinction painfully simple in Jira:

1. **Was the work engineering-driven?** Create a task.
2. **Was the work product-driven?** Create a story; use sub-tasks to create the work the engineers will do to complete that story.

Everything lives on the same playing field and is up for grabs to assign priority. So rather than bucket 20% time for debt, if all engineering-driven tasks (whether they be credits, debts, or something else) are the most important work to be done this week, they take up 100% of the sprint. Similarly, if product's work is the most important, it can dominate or completely own a sprint.

The important thing is that **the owner of the ticket sells why their work should be as important as it is**. If you can't sell why a critical security upgrade is vital to the health of your application then you need to learn how to sell and influence others. But then again, you might think that switching from Formik to React Hook Forms would be awesome but if it's based on personal bias and no tangible value creation for developer productivity you're dead on arrival.

So I encourage all of my engineers to read sales books. Learn how to sell and influence others. Convincing someone that what you want to do matters is a much better message than starting things off on the wrong foot by calling your work debt that needs to be repaid. Debt makes people wince. Creating value makes people smile. Paying down tech debt adds value so lead with that and focus on why your work is important instead of giving it the Scarlet Letter from the onset.
]]></content:encoded>
      <dc:date>2020-11-16T23:31:00+00:00</dc:date>
    </item>
    <item>
      <title>Tips to level up as a new engineer</title>
      <link>https://www.adamconrad.dev/blog/tips-to-level-up-as-a-new-engineer/</link>
      <description><![CDATA[My recent one-on-ones with my more junior engineers revolved around the same topic: how do I level up quickly to be a better aid for my tech lead?

]]></description>
      <pubDate>Fri, 30 Oct 2020 18:02:21 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/tips-to-level-up-as-a-new-engineer/</guid>
      <content:encoded><![CDATA[My recent one-on-ones with my more junior engineers revolved around the same topic: **how do I level up quickly to be a better aid for my tech lead?**

New engineers seem to deal with this dichotomy that they both want to learn quickly from the more senior engineers while also ensuring they bother them as little as possible because they are so busy and focused on big, important projects. While that is a reasonable line of thinking there are ways of achieving both without having to wait for the right opportunity to reach out to the top brass in engineering.

I've shared these tips over and over again so see if one (or all) of these tips resonate with you.

## Keep a daily log

One of the most common complaints I hear from more senior engineers is that their tips and suggestions for the more junior engineers just don't stick. It's like they are talking to a wall and the newer engineers just keep making the same mistakes over and over again.

This tip is pulled from the course [learning how to learn](https://www.coursera.org/learn/learning-how-to-learn) which I took quite a few years ago and it's all about **writing down everything you learn in your own words.** This serves two purposes: retention and recollection.

If you can summarize in your own words what you just learned then it is more likely you will retain that knowledge because you were able to translate it from someone else's words into your own. Further, if that retention degrades at all you always have your written log to search from which is your first line of defense to check against before you go asking senior engineers for help.

An added bonus for a daily log is that you can use it to catalog all of your successes when doing annual reviews. Skimming the last year's worth of notes will help you quickly recollect the highlights of your accomplishments in the last year and I bet you'll find some great wins you forgot about from the beginning of the year.

**The most important thing is that you start.** Even if it's just one sentence a day of something you found noteworthy. As long as you are creating the habit of maintaining a daily log then you can always fill it out as you add more entries and solidify the ritual.

## Cluster your questions

An old [post](https://signalvnoise.com/archives2/the_science_of_interruptions.php) from the woke productivity gurus at Basecamp mentioned that every time an engineer is interrupted it takes them 25 minutes on average to get back into the groove. That means fewer but more substantial interruptions are a net benefit to small, frequent interruptions.

You may think that just popping in on a Slack DM to a staff engineer to ask a 30 second question is an easy ask. The reality is that 30 second question, by this definition, is really a 25 minute and 30 second ask. Ten of those questions could easily add up to half a day of lost productivity for our poor staff engineer.

So instead of frequent interruptions you would be best served to your senior staff to cluster your questions together and take up more of their time in one swoop. Actually put time on their calendar for 15-20 minutes to just as a barrage of questions.

These questions can be unrelated or they can be part of the [five whys](https://en.wikipedia.org/wiki/Five_whys) of just one line of questioning. The collection honestly doesn't matter; what matters is that you are reducing the amount of 25 minute ramp-up times it takes for the senior engineer to get back to being productive.

The important thing is here is that **one large ask will always be an overall smaller commitment and pain than lots of small, frequent asks**. As a bonus, the cost is even lower if you keep a daily log and can reach for your own answers first before having to reach to someone else first.

## Review your own PRs

Whenever you post a pull request to GitHub, Gitlab, or whatever repository tool you choose, review your own PR first. There's usually a page before you hit that submit button where the scope of the `git diff` is posted in its entirety. That is a great opportunity to **be your own reviewer** and imagine your are your own tech lead ensuring your code looks great. This not only gives you more code reviewer practice but I'm fairly confident you'll find something yourself before you hit that submit button. This will not only save you a review round if the reviewer doesn't approve but it will also improve the perception that the quality of your code is great from the start.

Doing your own work upfront shows your colleagues that you sanity checked your own work and that you're putting your best foot forward. Most importantly, you'll save everyone involved in your review precious time.
]]></content:encoded>
      <dc:date>2020-10-30T18:02:21+00:00</dc:date>
    </item>
    <item>
      <title>Don't be clever</title>
      <link>https://www.adamconrad.dev/blog/dont-be-clever/</link>
      <description><![CDATA[I made a pretty frustrating mistake recently.

]]></description>
      <pubDate>Wed, 23 Sep 2020 22:09:22 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/dont-be-clever/</guid>
      <content:encoded><![CDATA[I made a pretty frustrating mistake recently.

I won't get into the details but essentially I was trying to do the right thing by creating a very clever set of circumstances to achieve a goal. The problem is that this clever setup had a gaping hole in it that was exposed by my People team. When I tried to resolve it with my VP, I realized that I could have solved this problem months ago with a simple conversation.

**Before you get clever, just talk with someone to see if you can get to your resolution in an easier way.** Sometimes all it takes is a phone call or a quick chat to get through the red tape.

I thought I needed to carefully route around all of the red tape to achieve my goal. I created a series of circumstances to check all of the boxes from all of the stakeholders to ensure I was obeying all of the rules we had set up for the problem I was trying to solve. I felt like I needed to fit within these rules to make an exceptional case work.

**The flaw I made was that I identified the exceptional case, and _didn't raise it as an exception_.** If you see an exception, you should _make an exception_. Instead, I tried to fit an exceptional case into the standard workflow by carefully creating all of these circuitous processes to ensure that both the exception was taken care of and I was staying within the rules of the road.

But when you have an exceptional case for something, whether it be hiring, promotions, or organizational design, it is now obvious to me that you need to just face it head-on rather than try to create all of this red tape to make it work within the system.

If we are a stickler for the rules we have setup without taking into account exceptional cases, we are doomed for pain and suffering. Being too dogmatic with our rules as leaders makes things too inflexible and brittle. On the flip side, we also don't want to allow too many exceptions to pass through or else they will, by definition, fail to be exceptional.

Finding the right balance of breaking, bending, and sticking to the rules is vital to ensure that the standards you set for your teams are a healthy addition to the processes you already have in place.
]]></content:encoded>
      <dc:date>2020-09-23T22:09:22+00:00</dc:date>
    </item>
    <item>
      <title>Architecture for managers</title>
      <link>https://www.adamconrad.dev/blog/architecture-for-managers/</link>
      <description><![CDATA[In a recent one-on-one with my boss he mentioned that, as a director, one of my core duties involves systems architecture. I was thrown for a loop a bit because while I’m happy to stay technical I didn’t think this was what many directors do. Yes, you need enough of an understanding of the systems you build to defend them in presentations with senior leadership or to help guide your teams to victory but I didn’t think it would involve things like architecture review and the kinds of systems design thinking you see in interviews. Though in just typing that I’m really glad I now have this muscle to flex daily because I know big companies put a focus on systems design for their management interviews.

]]></description>
      <pubDate>Mon, 24 Aug 2020 22:22:22 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/architecture-for-managers/</guid>
      <content:encoded><![CDATA[In a recent one-on-one with my boss he mentioned that, as a director, one of my core duties involves systems architecture. I was thrown for a loop a bit because while I'm happy to stay technical I didn't think this was what many directors do. Yes, you need enough of an understanding of the systems you build to defend them in presentations with senior leadership or to help guide your teams to victory but I didn't think it would involve things like architecture review and the kinds of systems design thinking you see [in interviews](https://github.com/madd86/awesome-system-design). Though in just typing that I'm really glad I now have this muscle to flex daily because I know big companies put a focus on systems design for their management interviews.

So what does systems architecture look for an engineering manager or director?

## Systems design

Of course I had to bring this up. I already linked to a big list of curated questions to think about in practicing systems design but I'd also point you to [this article](https://robertheaton.com/2020/04/06/systems-design-for-advanced-beginners/) from a few months ago which sort of builds on the basics. I know I need to study these more but I do have a few architecture examples that seem particularly relevant to the domain I work with in my day job.

## Understand your domain

Speaking of understanding the domain you need to ask yourself "how can my business domain operate within the context of the technical solution we've built?" In my case we work in logistics. So how can engineers, designers, product managers, and logistics coordinators all operate within the context of the supply chain software we've built?

That means I need to understand the business domain from all angles and all consumers of this data: from the employees running the dashboards and admin software to our drivers who power the trucks that move our goods across the country. The domain we operate in must be understood to provide an adequate technical solution.

## Understand your business

I already spoke about business domain but let's zoom out to the entire business: even if you have a solution for your customers and the employees who work on that software, how do you fit this solution into the greater context of the company you work for? How does the culture of software in your business adopt a solution like the one your teams are building? If you're working in B2B and there's a big sales culture with heavy investments in Salesforce you're going to need to find a way to integrate into Salesforce because building your own soltuion (or ignoring it) just won't cut it. As much as we pride ourselves in tech that we can build anything, we can't forget that our **software does not exist in a vacuum** so the systems we integrate must work within the context of the business we are a part of.

## Be mindful of velocity

Now that you know who you're building for and the context you're building solutions in, you need to build something that will scale _quickly_. I can't think of a single company that would like you to take your time building out the perfect solution at the expense of capturing customer revenue. If you're going to build something you can't just be thinking about technical scale like horizontal or vertical scaling. You also have to think about developer scaling: will this design decision increase or decrease the velocity of my teams? Can we build our solutions in such a way that **anticipate future needs** and **allow everyone to do their jobs more efficiently?**

One way to make your teams faster is to build a UI pattern library so UI developers can re-use components without having to build them from scratch. Another way to accelerate teams is to build an admin dashboard for your sales reps to complete transactions on behalf of customers who get the [white glove experience](https://snapshotinteractive.com/deliver-white-glove-treatment-client-part-1/), complete with validations and error checking so they get the transaction right the first time with a better chance of accepting payment.

While the second feels like regular old product work, it is still an improvement in velocity. You know that sales reps are going to push hard to complete sales so anything you can do to get to the sale faster with less errors is going to accelerate your whole company (even if they aren't developers).

## Start with the ideal state and work backwards

With all of this in mind you want to start by building a picture of your ideal state of your applications. What does the future look like if you could snap your fingers and build your ideal solution today? How does it handle millions of users executing billions of transactions flawlessly? What systems and integrations are required to achieve this beautiful feat?

Now take that ideal world and work your way back to today. What is required to get from Point A to Point B? More importantly, _when_ is it required? We don't want to pre-optimize or over-engineer solutions that we don't need. It is important to identify landmarks of scale so we know when to switch from Phase 1 to Phase _n_ throughout our architectural journey.

Once you have that road map, you have begun to build a complete picture of the systems architecture for you software from today to the next several years.

## Building the solution that meets everyone's needs

It's no small feat to create something scalable, fast, and works for both employees and customers. But then again, the job you're in isn't meant for everyone either! It took you years of practice, failure, and learning to get to a level of management that we're at today. To continue to push and improve, we have to be thinking even bigger than we thought yesterday. I know I have a lot of homework to do over the next several weeks.
]]></content:encoded>
      <dc:date>2020-08-24T22:22:22+00:00</dc:date>
    </item>
    <item>
      <title>Accountability is not responsibility</title>
      <link>https://www.adamconrad.dev/blog/accountability-is-not-responsibility/</link>
      <description><![CDATA[I’ve noticed some of my colleagues seem to use the terms accountability and responsibility as if they were the same.

]]></description>
      <pubDate>Wed, 19 Aug 2020 00:05:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/accountability-is-not-responsibility/</guid>
      <content:encoded><![CDATA[I've noticed some of my colleagues seem to use the terms _accountability_ and _responsibility_ as if they were the same.

They are not. I'll tell you a story to illustrate the difference.

When I was a kid my best friend and I were inseparable. In the summers, not a day could go by without one of us going over to each other's house.

One day I was over his house when my mom came over to pick me up with my younger brother. Since my best friend and I were inseparable you can understand that my mom was also friends with my best friend's mom so she didn't just drive up to the driveway to get me. She got out of the car, said "watch your brother" and started chatting with the family.

While our moms were distracted catching up, the boys were focused on a game of hit the ball over the fence. I should mention we were pretty young: I don't think I was much over the age of 10 which means my brother was closer to 5-6. Young enough not to know better. So when I tossed the ball up in the air to knock it out of the park, my brother came fast approaching.

On my back swing before hitting the ball, I instead hit my brother in the jaw with the tip of my metal baseball bat. The thud instantly caught the attention of my mother and she reached to comfort my brother. You know an injury is bad the longer it takes for a child to cry. Bumped your head on the coffee table? Instant tears. Hit with a baseball bat (unintentionally, of course)? There's a good 10 seconds of processing what happened followed by ear-shattering screams.

First thing my mom did was ask me what happened? As the good boy scout I couldn't possibly tell a lie so I told her exactly what you read: I swung the bat backwards, didn't see my brother, and his face met my bat. It was my fault; I didn't look before I swung a giant metal bat.

## What does this teach us about accountability and responsibility?

It was my **responsibility to look out for** my brother. I was responsible for his well-being and thus I was responsible to look around to ensure it was safe to swing a bat.

Meanwhile, I was **accountable for the injury**. The who, what, when, where, why all fell on me: I did the crime, I didn't check my surroundings, and my brother paid the cost.

The easiest way to distinguish the two is to know that **responsibility is for the future and accountability is for the past**.

When you think about how you lead know that you are **responsible for the productivity, effectiveness, and well-being of your teams**. There's a lot of "who" and "when" tied to responsibility: taking great people and helping them achieve things in their future.

When you think about projects you've accomplished or failed then you are **accountable for the result**. As I've seen called out time and time again, you give credit to everyone on your team when you succeed and you take it on the chin yourself when you fail.

## Why do you need to know the difference?

In leading by example and teaching others we need people to understand that **when our future leaders gain responsibility they also gain the outcome of those new duties**. A young and ambitious engineer is promoted to senior with a big pay bump and fresh title. It's easy to get on their high horse and relish in the pay and status.

But a senior title now comes with on-call duty 24/7 for one week every 2 months. Their code is subject to more scrutiny and demands a higher level of quality. Fewer mistakes are tolerated. You have to earn your keep. And now that this engineer can become a tech lead they also have to answer to the boss if the project fails. Before they could hide in the shadows if a project was delayed. Now they are front and center to answer if something is delayed or broken.

Aim to increase your responsibility. Just know that this comes with more things to answer for if things go wrong.

]]></content:encoded>
      <dc:date>2020-08-19T00:05:00+00:00</dc:date>
    </item>
    <item>
      <title>ABF - Always Be Failing</title>
      <link>https://www.adamconrad.dev/blog/abf-always-be-failing/</link>
      <description><![CDATA[If you want to get ahead in life you need to fail. Failure is growth.

]]></description>
      <pubDate>Tue, 18 Aug 2020 00:19:06 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/abf-always-be-failing/</guid>
      <content:encoded><![CDATA[If you want to get ahead in life you need to fail. Failure is growth.

Getting everything right teaches you nothing but to be smug. Failure keeps you humble and feasting for ways to improve.

Here are a few recent examples and what I learned:

## Improving hiring

I _hate_ to lose. So you can imagine how much my day was ruined when last Friday I received an email from a candidate I was really looking forward to hiring that they would not be choosing us.

I thought we had this one in the bag but thankfully the candidate gave us a reason for why they didn't go with us.

If we had landed this candidate I might begin to get complacent. I've had the joy of hiring a lot of great candidates in a short amount of time. I've personally hired nearly two dozen people in just the last year I've been running my teams at Indigo and there is room for me to believe I am good at hiring. This would be a great opportunity for [confirmation bias](https://en.wikipedia.org/wiki/Confirmation_bias) to come out of the woodworks.

Instead, this was a huge hit to my ego. What am I doing wrong? I thought I figured this candidate out; how could I be so blind? Why the hell does this seem to happen to me at crucial points where I need a yes?

I tend to obsess about certain thoughts that ruminate in my brain for too long. But in asking myself so many questions I knew that I had to have follow-ups with teammates. **Post mortems are a great opportunity to learn from failure.** So I chatted with the key folks in this hiring play. What can we learn here? How can we make changes _today_ to fix this for future candidates so we don't lose them?

**None of this questioning happens when you win. You only try to refine and perfect when you lose.**

## Over communication

I've got a lot of new teammates with a lot of awesome energy to make a great first impression. With so many new folks and so many new ideas thrown around it can be difficult to follow which ideas are canon and which are breaking new ground.

So I was surprised to find out that I had a case of a policy a developer thought was adopted that had never existed before.

I won't get into details but this would have been a slippery slope kind of policy if not controlled properly. It has amazing roots but without getting into details it was a good policy that could easily be abused and misinterpreted if not carefully communicated across the organization.

This was an eye opener about the often touted advice to [over communicate](https://adamconrad.dev/blog/technical-lead-management/); share the same message over and over again because you can never guarantee if your message was received in the way you intended it to.

In this case, I was extremely confident that the message was received loud and clear. But this suggested policy was a clear misinterpretation of what I had said. Whose fault is that?

I'm a big believer in [extreme ownership](https://www.amazon.com/Extreme-Ownership-U-S-Navy-SEALs/dp/1250067057) so I immediately wondered what _I_ had done (or not done) to miss the mark communicating my ideas. How could my ideas be interpreted this way to shape policy when they shouldn't have? Why had I not clearly communicated to all parties our stance on our policies in software development?

Again, **I had an inner dialogue with myself asking so many times why I failed** until I got to a root cause. The answer? Suggestions need to be labeled as suggestions. Leaders have influence and people will take your word quite literally. **If you don't specify otherwise your words as a leader may be considered the unequivocal truth.**

Communicate your message. Then reiterate it on Slack. Then say it again in an email one. And maybe say it one more time on your Wiki. Communicate until everyone on your teams not only hears your words but understands their meaning as well.

## Common themes

So from these two examples we can draw some conclusions on failure:

* **Failure leads to questioning.** We don't question when we win. Our brain goes _wild_ when we lose. There's an instinct to reject failure and put the blame elsewhere. But that instinct in a way is a form of questioning: why didn't the other person land that candidate? Why can't my company offer our employees this benefit? These are still questions. And the answers are opportunities to grow.
* **Questioning leads to problem solving.** There's no problem to solve if you succeed. As engineers we love to solve problems so in a way failure is something we crave.
* **Solving problems is fun and educational.** Every time I've failed I've gone through this process of questioning to make sure it never happens again. Sometimes it does but most of the time I don't repeat the same mistakes. That is improvement and accomplishment. Stack enough of those over a career and you have a resilient person with a lot of knowledge and wisdom.

The more failures you have the more growth you have. And the more growth you have the more chances you have to succeed big when opportunity comes knocking.
]]></content:encoded>
      <dc:date>2020-08-18T00:19:06+00:00</dc:date>
    </item>
    <item>
      <title>3 basics of TypeScript you probably do not know</title>
      <link>https://www.adamconrad.dev/blog/3-basics-of-typescript-you-probably-dont-know/</link>
      <description><![CDATA[I’ve been programming TypeScript professionally for the last two years. Even with that much TypeScript experience it’s always useful to have a beginner’s mindset.

]]></description>
      <pubDate>Sun, 26 Jul 2020 12:19:06 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/3-basics-of-typescript-you-probably-dont-know/</guid>
      <content:encoded><![CDATA[I've been programming TypeScript professionally for the last two years. Even with that much TypeScript experience it's always useful to have a [beginner's mindset](/blog/always-have-a-beginners-mindset).

So it's no surprise to me that I learned a few things from watching [this tutorial video](https://www.youtube.com/watch?v=BwuLxPH8IDs) on TypeScript basics. Here are my favorites which somehow never really stuck the last few years.

## Tuples are a thing

Array types are pretty straightforward in TypeScript. If you wanted an array of numbers you simply use the type `number[]` with your new variable. But what if you knew exactly what kind of array you're going to use? What if you know the array is of fixed length and of fixed types?

Enter the **tuple**. In other programming languages, a tuple is defined as a _finite sequence of terms_. One common tuple we use are enumerations. Enumerations have string variables as their key, and numbers as their value.

In TypeScript, you could describe enumerations as `[string, number]`, but they are so common there is an explicit `enum` keyword to encapsulate this concept.

A more practical example would be a latitude/longitude combination. Libraries like Google Maps will return a tuple of lat/long numbers. Rather than pull out those values into a `number[]`, you know it is specifically a number array of only two values, so `[number, number]` would be more appropriate here.

Why does this matter? Because if you have a `latLng: number[]` and you `push` a 3rd value, TypeScript will not complain. But if you have a `latLng: [number, number]` and you `push` a 3rd value, TypeScript _will_ complain that your array is no longer a part of your specified type. And since lat/long pairs only come in pairs, you know that 3rd `push` was an accident.

Another example would be 3D graphics programming. Every point in 3D space has to fit within the `x`, `y`, or `z` axis. So you could declare `type Point3D = [number, number, number]` so that all points in space fit into this tuple (e.g. `const startingPoint: 3DPoint = [0,0,0]`).

## You probably meant `unknown` instead of `any`

We should avoid `any` in our code because that's TypeScript's way of throwing its hands in the air and saying "I give up." To which I would say, "why are you bothering to use TypeScript"?

Now that I know what `unknown` is used for, I now look back on every use of `any` and now realize it probably could have been replaced with `unknown`.

`any` says it can be any type: `number`, `string`, `boolean`, `object`, `Foo`...whatever you want. And it will forever stay that way even if you mutate it.

`unknown`, on the other hand, allows for any type because you don't know what you're going to get _yet_ but we know we will figure it out later.

The best example I can think of is again the use of external libraries like Google Maps. The chimera of objects you get on the way in may be complex or variable but you know you're going to coerce some of those objects into numbers for some kind of plotting onto your own map canvas. Because you have knowledge of the future transformation into something defined, `unknown` is a better type to use here than `any` because it will become a number and won't change.

If you're struggling at times to adhere to the [`noImplicitAny`](https://www.typescriptlang.org/docs/handbook/compiler-options.html#compiler-options) rule, try working with `unknown` before completely giving up on resorting to `any`.

## `never` is great if you plan on throwing errors

Finally, there's a strange little type in TypeScript called `never`. I haven't used this one yet and it's because I've yet to find a reason to use it...until now.

`never` is a type that tells TypeScript "a value will never be returned." And while JavaScript has no concept of this (and will default to `undefined` when nothing is returned) the subtle distinction between nothing returned and never returning is best summarized with error handling.

Let's say you're writing some custom error handling functions to purposefully `throw` an error and break the app. In this case, you can't complete the function. It's not that the return value will be `undefined`, it's that the program stops executing altogether and you _never_ get that far.

You probably won't be in a situation where you need to _only_ throw an error (i.e. a `try`/`catch` would be a better pattern) but this type provides this opportunity.

### Bonus tip: `void` and `undefined` are different but you probably won't use it

As a bonus tip, I wanted to talk a bit more about `void` which was a strange keyword to me at first because of the fact that it reminds me of [`void` in JavaScript](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/void) which I avoid whenever possible.

Along the lines of `never`, `void` is a similar type to describe the concept of nothing being returned. JavaScript does have this concept because nothing being returned defaults to `undefined`. But if a function can return `undefined`, how are these two examples different?

```TypeScript
// how are these two different?
const noReturnFn: () => undefined;
const noReturnFn2: () => void;
```

The answer is that to JavaScript, there is no difference, but to TypeScript, it's all about if you `return` or not.

```TypeScript
const noReturnFn: () => undefined = () => {
  console.log('This must return');
  return;
};
const noReturnFn2: () => void = () => {
  console.log('This must not');
};
```

As you can see, a function return type of `undefined` requires that you use the `return` keyword while `void` does not return anything explicitly.

If we go back to our [compiler options](https://www.typescriptlang.org/docs/handbook/compiler-options.html#compiler-options) we see a suggested code quality option of `noImplicitReturns`. If we turn this on then `void` becomes a useless type because you are no longer allowed to return nothing; you _have to_ explicitly return something with all of your functions. Now in situations where you would use `void` you'll have to default back to `undefined`. Personally, I think this is better because you are being more intentional with your code but now you at least know why `void` exists in TypeScript and when you would use it.
]]></content:encoded>
      <dc:date>2020-07-26T12:19:06+00:00</dc:date>
    </item>
    <item>
      <title>Don't nickel and dime offers</title>
      <link>https://www.adamconrad.dev/blog/dont-nickel-and-dime-offers/</link>
      <description><![CDATA[It’s so hard to find great engineers that pass all of your interviews with flying colors. So when you do find someone who actually makes that happen, don’t let them go because you feel the need to bargain over little bits of money.

]]></description>
      <pubDate>Mon, 20 Jul 2020 22:23:58 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/dont-nickel-and-dime-offers/</guid>
      <content:encoded><![CDATA[It's so hard to find great engineers that pass all of your interviews with flying colors. So when you do find someone who actually makes that happen, don't let them go because you feel the need to bargain over little bits of money.

One issue is you may lose them because you're perceived as a company that wants to haggle down to the last dime. If they'll haggle over that, how do you know they won't treat you well once you arrive? Do you _want_ to work for a company where they are going to nickel and dime your every interaction?

These are the questions candidates will ask if you can't come to a reasonable negotiated offer quickly. Sure, coming up or down by $10k or more makes sense and is part of every negotiation. I find myself impressed that candidates are doing that more often these days. But if you're closed in on, say, less than $5k in differences, let it be. You won't miss that money.

My gut instinct at first is if a candidate is negotiating hard, you don't want to seem like your organization is soft and that you'll curtail to any demand an engineer asks for. This happened to me today and when I took a step back I thought "well what is the alternative? Do I really want to let this candidate go for a few grand?" The answer was certainly _no flippin' way_.

So remember: negotiation is good. Haggling down to the last few thousand dollars is not. You've probably already invested that money in just assembling the people to interview this candidate in the first place.

Oh, and when you are ready to make an offer, *sync up with your recruiting team to ensure the benefits are up-to-date*. I paid for this one dearly last week; turns out our benefits were _better_ than we advertised and we failed to mention that and lost a candidate. Save yourself that grief and make sure you know your benefits are current so you're putting your best foot forward.
]]></content:encoded>
      <dc:date>2020-07-20T22:23:58+00:00</dc:date>
    </item>
    <item>
      <title>How to practice frontend engineering</title>
      <link>https://www.adamconrad.dev/blog/how-to-practice-frontend-engineering/</link>
      <description><![CDATA[I’m a big fan of Will Larson’s blog so his recent post on practicing backend engineering inspired me to write an equivalent article for frontend given my previous experience in that space before moving into management.

]]></description>
      <pubDate>Sun, 21 Jun 2020 11:38:34 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/how-to-practice-frontend-engineering/</guid>
      <content:encoded><![CDATA[I'm a big fan of Will Larson's blog so his recent post on [practicing backend engineering](https://lethain.com/how-to-practice-backend-engineering/) inspired me to write an equivalent article for frontend given my previous experience in that space before moving into management.

I will say one thing that makes it easier to practice frontend engineering is that everything can happen within the browser. You don't even need a site with real data to practice! So what does frontend practice look like?

* **Writing HTML templates.** You need some sort of document to express your design's content.
* **Designing with CSS.** The look, feel, color, and experience is most uniquely expressed via the design controls within your style sheets.
* **Making pages interactive with JavaScript.** As clients have become more interactive there has been more and more reliance on frontend JavaScript frameworks to build these client-side application as quickly as possible.
* **Building infrastructure to support the above three.** Things like linting, schema generation, UI library building, lossless image compression, and asset packaging are part of the piping required to make the previous three run smoothly.

## Rebuild the design of your favorite site

The absolute easiest, no-brainer way to practice frontend is to just pick your favorite big site like Facebook, Twitter, or Stripe and re-implement the pages you see. See how close you can get to pixel-perfect renditions of your favorite sites. Nail the CSS, the JavaScript interactivity, and the semantic HTML. See what libraries and plugins they use. And the best part, **you can just view source to check all of your work**. Unlike backend engineering, the answers are right in front of you. If you're ever stuck or wondering how they did something, all of the code is available right in your browser client.

## Create a blog or write your own static site generator

So many code schools or pet projects have you building a Twitter clone or some hobby toy project you will never use again. If you build your own blog using your own static site generator, **there's you're homepage**! What a better way to practice frontend than to actually build something you will really use if for not other reason than to showcase as your homepage your frontend skills!

Make your design as fun and whacky as possible. Use fringe, bleeding-edge CSS tools. Write about your experience building the blog _on your blog_.

## Whatever project you choose, make sure you test yourself on these things

It's not enough to simply copy a design or write whatever you want. There are a few ways to separate your code from that of a professional.

* **Have it pass a bunch of audit tools.** [Google Lighthouse](https://developers.google.com/web/tools/lighthouse/#psi), [YSlow](http://yslow.org/), [aXe](https://www.deque.com/axe/), and [React Developer Tools](https://reactjs.org/blog/2018/09/10/introducing-the-react-profiler.html) are all examples of ways to ensure your code is fast, accessible, and easy for crawlers to read.
* **Make it responsive for every size.** Use the principles of [mobile-first responsive design](https://www.uxpin.com/studio/blog/a-hands-on-guide-to-mobile-first-design/) to create a site that works on any screen size. You should be able to grow and shrink your screen on your desktop (or phone) and have the site look beautiful no matter what random width you have set.
* **Run your code against linters and formatters.** Make liberal use of [Prettier](https://prettier.io/) and [ESLint](https://eslint.org/) to write well-formatted, easy-to-read code. Understand the rules and why you would use them.
* **Use a bundler and packaging tool.** Most sites use something like [Grunt](https://gruntjs.com/), [Gulp](https://gulpjs.com/), or [Webpack](https://webpack.js.org/) to efficiently package their assets for browsers to consume. Even if it might be overkill for your blog, it is still useful to get practice using it and optimizing your bulids for extreme performance.
* **Really understand every tab of Chrome/Firefox DevTools.** What would a memory leak look like? How could you tell if there's a performance problem on the client or the server? Can you spot [runtime issues](https://developers.google.com/web/tools/chrome-devtools/rendering-tools/) in painting, rendering, composing, and scripting? DevTools can give you all of these answers if you know where to look.

If you can nail these points, you will be thinking not just like a hobbyist, but a professional frontend developer.

### Throw it all away and do more with less

And when you managed to get everything passing perfectly with all of the practice above, start over with as little as possible. There's a reason I love [this website](http://motherfuckingwebsite.com/). It does _exactly_ what it says it will do and nothing else. Sometimes I think we've forgotten what the web was really intended for. It is first and foremost a document store and linking tool. We use the web to share information. The hyperlink is the glue that connects these documents to create a literal web of networked pages.

The humble `<form>` is the purest representation of interactivity between your browser and the rest of the internet. Between documents and forms, that's really all we need. Everything else is just a pretty distraction designed to keep you clicking and engaged. If we remember the roots of what we are building we tend to forget the noise of the latest SPA framework or hip new bundler and instead just focus on information architecture and user experience. Don't lose sight of that.

## Additional resources

Beyond hobby projects, I tell most people they should just build a real SaaS app and scratch their own itch. Push something real to production that you can charge customers for. This will have you not only practicing frontend, but backend engineering as well. More than you bargained for but there is truly no better way to practice your craft.

To create front ends that scale, I'd recommend a few more resources that I've found incredibly valuable over the years that stand the test of time:

* _[Front End Architecture for Design Systems](https://www.amazon.com/Frontend-Architecture-Design-Systems-Sustainable/dp/1491926783)_ - This book is the reference I use to cover architecture for even the biggest production apps I work on. Though some of the tools and technologies are dated, the tenants of architecture remain indelible.
* [MDN Web Docs](https://developer.mozilla.org/en-US/docs/Web) - The gold standard for understanding everything web technologies. This is easily the most comprehensive guide and reference for all things HTML, CSS, and JavaScript. They have tutorials, reference material for every property and function in all three specs, as well as a host of great examples and polyfills. I probably reference this at least once a week.
* [Google's Web Fundamentals](https://developers.google.com/web/fundamentals) - These may be fundamental but they are by no means easy or rudimentary. I still find plenty to learn even with more than a decade of experience. These documents always update to include the latest and greatest web technologies as well.
* [Nielsen Norman Group](https://www.nngroup.com/articles/) - There is no better team to learn user experience from. This UX firm has authored some of the best selling books on the topic are considered the foremost experts in their field.
* [Refactoring UI](https://refactoringui.com/) - This book, and their [free collection of tweets](https://refactoringui.com/) has single-handedly made me a better designer more than any other resource. If you want to create beautifully-designed sites with the tools you already use, you _have_ to check out this book and site.
]]></content:encoded>
      <dc:date>2020-06-21T11:38:34+00:00</dc:date>
    </item>
    <item>
      <title>Organizational design for marketplace companies</title>
      <link>https://www.adamconrad.dev/blog/engineering-orgs-for-marketplaces/</link>
      <description><![CDATA[I’ve worked at four marketplace startups in my relatively short career. I do at least have enough experience on how I’ve seen successful (and unsuccessful) companies scale their engineering organizations for marketplaces. There is a unique problem of scale that only happens with marketplace startups and it is worth addressing as a post. If you are an engineering leader for a growing marketplace startup, here are a few tips to handle team growth in many stages.

]]></description>
      <pubDate>Tue, 21 Apr 2020 14:09:19 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/engineering-orgs-for-marketplaces/</guid>
      <content:encoded><![CDATA[I've worked at four marketplace startups in my relatively short career. I do at least have enough experience on how I've seen successful (and unsuccessful) companies scale their engineering organizations for marketplaces. There is a unique problem of scale that only happens with marketplace startups and it is worth addressing as a post. If you are an engineering leader for a growing marketplace startup, here are a few tips to handle team growth in many stages.

## Why are marketplace startups special snowflakes?

If you follow [Andrew Chen's](http://andrewchen.co/) musings on marketplaces, you'll see there are some unique dynamics at play. The most obvious difference with most companies is that there are **two primary personas to concurrently manage**: supply and demand users. Both are needed to keep a marketplace going so both require the utmost attention and care. A company like Facebook only needs to keep the general user happy. But Airbnb needs to make sure that home owners have a great experience listing their homes to keep supply up _while also_ ensuring that demand is there to keep houses filled and that these customers pay in a timely manner while feeling safe being in someone else's home.

This need to please two personas instead of one presents unique challenges because right out of the gate you're going to need to have two products under one umbrella application: one for buyers and one for sellers. So do you fracture your team into two from day one? Let's dive in.

## At first, focus on supply

When your engineering team is small enough for just you to manage (anywhere from 8 to 12 depending on how masochistic you are), you should focus on supply. Why supply? Because no demand customer wants to enter a marketplace with nothing to purchase. It is far worse to flood a site with demand customers and have nothing to buy than have supply customers with nothing to sell.

If your marketplace is novel, then your supply customers are already unable to sell the thing they are now trying to sell on your marketplace. So if they create a profile and no one buys they are no worse off than they currently are, which means they can tolerate having a dormant profile for a while. **Your marketplace provides pure upside to their life which is increasing their revenue, so they will tolerate a subpar or empty experience to start.**

Demand customers, however, do not have that same tolerance for a subpar experience. If they go to your site to find 0 vendors able to fulfill their needs, they will leave and possibly never come back. You cannot mess up this relationship, so it is vital that you give them what they want from day one, which means flooding your site with ample supply. So cater to your supply-side of the marketplace first and you will have plenty of room to scale for marketing and sales when those channels can finally start acquiring demand.

## Once supply hits MVP status, use that same team for demand (or hire for it)

As soon as you have something [minimal but loveable](https://firstround.com/review/dont-serve-burnt-pizza-and-other-lessons-in-building-minimum-lovable-products/), you can start to focus on the other side of the marketplace. It's very difficult to raise money on a concept, so my guess is once you have your MVP for the supply-side you're still small enough (i.e. a handful of engineers in your flat org) that you haven't hired enough people to warrant to dedicated product teams. So instead, **put your supply-side app on maintenance mode and start all over again with your demand-side of the marketplace.**

To reiterate, your supply-side users see your app as pure upside, so they are willing to tolerate a lot if you provide them value. So if you give them a great starter experience, you don't need to continuously reinvest in their success (yet). For now, squash bugs, make the experience liveable, and then move on to completing the marketplace with the same team. This initial team will be the core owners of your product, and now is the time you should start looking towards succession planning to find these key early leaders who can start to run the two teams once your org is big enough to support dedicated teams to each side of the marketplace.

### Your internal teams are going to want to play around as well

As a corollary to this shift for your engineering team, you're going to need to also think about the internal users (e.g. customer experience, account managers, PM, and design) who will want to play with the app as well. You'll want super user accounts, an admin panel, and lots of other back office tools to support the marketplace under the hood.

While I highly encourage teams to [dogfood their own apps](https://newrepublic.com/article/115349/dogfooding-tech-slang-working-out-glitches) instead of prioritizing internal admin dashboards, you're going to need to account for both kinds of interactions. My advice here is to **build these functionalities only when absolutely necessary**. When someone at the company cannot get their job done because there isn't a tool for it, then build it. Until then, focus on your supply and demand customers.

## Next fund raise event, add that second team

Now you have a true MVP with a dozen engineers or so. What's next? If you could snap your fingers and magically hire a dozen more engineers, you'd have two teams dedicated to each side of the marketplace. But with limited resources and the cost to train and hire folks, this isn't going to happen anytime soon. So instead accept that **the needs of your marketplace customers will constantly oscillate in priority between supply and demand**.

I've seen it happen in every single marketplace company I've worked at: you build something for 6 months for a supply-side customer and next thing you know the next two quarters are completely devoted to the demand-side customer with senior leadership wondering how we could have been so negligent to this cohort for so long. Fast forward another six months and you'll hear the exact same message about the supply-side all over again. You're never completely ignoring either side, you just end up dramatically shifting the priority structure based on your team resources and the marketplace economics that are present with your product.

I haven't said much about the actual engineering organizational design at this point because there really is none. It basically looks like this:

```
You
+-- The 8-12 engineers that work for you
```

At some point you'll grow large enough that you simply cannot handle another direct report. If you followed my advice from above, you should have found your star performer who hopefully has a knack (and desire) for management and is willing to run another team. At that point your _n_ th engineering hire (the one where you decide you can't manage their career anymore) is the catalyst to divide the team into two:

```
You
+-- 6 of the engineers to stay on Team Alpha
+-- Star engineer/manager of Team Bravo
|   +-- 6 of the engineers to join Team Bravo
```

I've left the division vague on purpose because every engineering organization is different in how they want to segment teams. Ideally you have 1 team for supply and 1 team for demand and it is up to you to choose which team you want to run versus your newly-minted engineering manager/tech lead.

What this does is allow for growth of both teams without neglecting either one. **With the understanding that you will need to accept some tolerance in which team gets more resources**.

Per my earlier point about the oscillating needs of your marketplace personas, sometimes you will need to shift teammates from one team to another because the needs of a particular side of the marketplace are so large that you need to swarm on important initiatives and move towards maintenance mode for the other side. In that case, you still have a team at the ready, but the larger team can be bolstered and scaled as desired by product or senior leadership.

At some point, the group will be large enough that your organization should put both you and your EM at the limit:

```
You
+-- 8-12 engineers on Team Alpha
+-- Star engineer/manager of Team Bravo
|   +-- 8-12 engineers on join Team Bravo
```

With an engineering organization hovering somewhere between 18-26 people (including yourself), things will start to get interesting. Depending on the company, the size of the whole outfit has grown anywhere from 50 to 100 people and you will have hopefully seen a few rounds of fundraising and numerous shifts in company culture. This is the pivot point where things have to be designed for scale.

## Finally, build a process for scale

If you've made it this far, your company is ready to prove product-market fit and you're going to have to think more about how you can scale your organization to meet the demands of your ever-growing customer base. I could give more advice on this topic but once you have enough this many engineers the nuances of marketplaces no longer matter since you can support both sides of the marketplace.

There are plenty of books and articles on how to design an organization for scale, so I'll offer some quick tips from how to get from two dozen to four dozen engineers to really fill out your marketplace engineering teams:

* **Platform engineering** (+8 engineers). These engineers will chiefly support your internal customers, who now include developers. They need help with a platform that can scale to meet all of these customer needs.
* **Infrastructure/DevOps/SRE** (+4-8 engineers). For smaller teams you can probably get away with Heroku or some DevOps-As-A-Service. At some point your operation gets large enough that you'll want dedicated folks who can support all of that infrastructure to scale on demand because the cost of hiring someone is cheaper than the cost to keep Heroku churning at a fast pace.
* **Mobile/Desktop engineering** (+8-16 engineers). If your MVP was on web, you probably need an equivalent mobile offering (or vice versa). Additionally, you may need to think in doubles since you'll have to support iOS first and then Android. The cycle for supply and demand MVPs for these device-specific products starts all over again.

As you can see, we've quickly gone from 26 to as many as 58 in your engineering organization. You'll now have _several engineering managers reporting to you_.

If you scale to all of these teams, you may have enough engineering managers that it is again too difficult to manage all of these relationships and you'll have to think about adding a Director of Engineering (or two) to split the managers with you. Again, I don't want to prescribe an exact rubric at this point because this topic has been covered numerous times, but if I were to build a marketplace engineering organization from scratch, this is the methodology I would follow.

Hopefully this was a useful exercise if you're a brand new CTO or VP of engineering that is working at a marketplace startup. If you found this useful but I missed something or you have questions on this approach, I'm always happy to chat on [Twitter](https://twitter.com/theadamconrad) or [HackerNews](https://news.ycombinator.com/user?id=acconrad) where I'm most likely to be talking about this stuff. Happy scaling!
]]></content:encoded>
      <dc:date>2020-04-21T14:09:19+00:00</dc:date>
    </item>
    <item>
      <title>Can you turn around a dysfunctional team?</title>
      <link>https://www.adamconrad.dev/blog/can-you-turn-around-a-dysfunctional-team/</link>
      <description><![CDATA[I recently read a post on the habits of high-functioning teams with a great discussion over on HackerNews. One person, in particular, wondered if you could apply these concepts on to dysfunctional teams and turn them around. Since I’ve have the opportunity to turn around teams before I thought I would weigh in on this question with each habit.

]]></description>
      <pubDate>Tue, 21 Apr 2020 14:09:19 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/can-you-turn-around-a-dysfunctional-team/</guid>
      <content:encoded><![CDATA[I recently read a post on the [habits of high-functioning teams](https://deniseyu.io/2020/05/23/habits-of-high-performing-teams.html) with a great discussion over on [HackerNews](https://news.ycombinator.com/item?id=23290943). One person, in particular, wondered if you could apply these concepts on to dysfunctional teams and turn them around. Since I've have the opportunity to turn around teams before I thought I would weigh in on this question with each habit.

Short answer: **yes, dysfunctional teams can turn around**. I can't say all of them can because sometimes teams are just set up to fail in certain organizations (I hope you are never on one of them). With that out of the way, let's dive into team one-by-one.

## Psychological safety and trust work on anyone

This one to me is just so obvious. **If you learn to trust people, they will give back by being their best selves.** No one wants to be infantilized (except for my infant, he doesn't have a choice). My stance is *if you made it through the barrage of interview questions, then we trust you innately to some extend to get the job done*. So then you should follow through on that and trust your team! And it can start today.

The first step is to listen. Start having regular 1-1s with everyone on your team, even the people that don't report directly to you. [Get to know](https://gettoknowapp.com) them. Learn about who they are outside of work. We're all humans with emotions, thoughts, feelings, hobbies, and interests; we're not just your employees. So learn about what makes us tick and you'll find the underlying motivators for high productivity.

Second, continue to listen, but publicly. Now that you have private 1-1s, you also need public retrospectives for growth and learning. This was suggested in the original article and I would say it's a good 1/5th of the responsibilities I have as a director is to synthesize the learnings from our retrospectives. **Grwoth and learning come from failure, so don't chastize it; celebrate it!**

Finally, keep the two things separate. Don't mix public and private conversations because people hate to be chided in public and sometimes would like more than just praise in private. If you start to blur the lines it makes people feel like you don't have their best interest at heart.

And that includes stuff way beyond feedback. Are you seeing someone get burnt out? Don't just sit there, give them a day off! Ask them to take a mental health day to recharge. The worst decision here is no decision. Each of these olive branches builds trust, which is arguably the greatest currency you have as a manager.

## Great hygiene starts with checklists

So we know that dysfunctional teams can start to trust each other if you begin by extending olive branches. Once you have trust, can you whip the team into shape? You can if you make it so easy they can't mess it up. That's where checklists come into play. In fact, there is an [entire book on this concept](http://atulgawande.com/book/the-checklist-manifesto/).

Checklists are great because if you just follow them, you get results. What kinds of checklists are useful to turn around teams?

* **Pre-commit hooks.** Forgot to run the linter? Didn't add sufficient test coverage? Catch them before they hit code review with a git hook tool like [husky](https://github.com/typicode/husky).
* **CI/CD gates.** Tools like Jenkins and CircleCI can help you bake in best practices like checking for sufficient test coverage, that all tests pass, and that you've audited your libraries to ensure no security vulnerabilities.
* **Commit templates.** Ideally you can add them to your [commit template](https://gist.github.com/lisawolderiksen/a7b99d94c92c6671181611be1641c733) so whenever someone spins up a new pull request they have to check all of the boxes to verify they have consciously agreed to adhered to your standards and best practices.
* **Post-mortems.** Break the site? Let's chat about it. Found a security bug? Let's take 20 and recap. Post-mortems, if we have psychological safety, are not about the blame game, but about the [5 Whys](https://en.wikipedia.org/wiki/5_Whys) and learning from our failures. See a recurring theme here?
* **Meeting notes.** Even meeting notes have a [template](https://www.atlassian.com/software/confluence/templates/meeting-notes). The more important thing is that you get into the habit of doing them so people don't waste time in meetings and have stuff to take away for later.
* **Retrospectives.**  Yes, retros also have [easy-to-follow](https://www.atlassian.com/software/confluence/templates/retrospective) templates for checking off all of the things you need to cover to start growing and learning on a weekly basis.

Hygiene doesn't have to be difficult. Just set some guardrails in place for your teams and soon enough they will make these habits automatic because they are drilled into their brains with these checklists.

## Get everyone to participate in the game

The next habit is around giving everyone an opportunity to experience all facets of your team running production software. The article refers to this as something akin to RPG experience points for leveling up your characters. Put it another way, it's about **succession planning**.

If your top engineer were to disappear in the Bermuda Triangle tomorrow, who would run your on-call systems? Who would know the innermost workings of the system architecture? And how exactly _does_ Kafka work, anyway?

Don't get caught without answers to any of those kinds of questions. Team rotations, pair programming, and [ownership registries](https://lethain.com/productivity-in-the-age-of-hypergrowth/) are crucial to ensuring you spread the load across the entire team and don't have a single point of failure. You wouldn't scale your systems with only one copy of your database, so why are you putting all of your knowledge into one senior developer?

The best way to implement this on a dysfunctional team is to just have an extended 1-1 with your top performers and get them to do a brain dump of everything they know. Then cross reference that with the rest of your team and see if there is overlap. For the places where you have no overlap, start looking for volunteers from the junior ranks. Good junior and mid-level engineers will chomp at the bit to gain more responsibility and up their skills.

## ABC - Always be communicating

The last point of high-functioning teams is they communicate with generosity and good intentions. This one is probably the hardest nut to crack with regards to dysfunctional teams. Again, if you start with trust, you're more likely to get open communication.

But dysfunctional teams that have trust still may be bad at communicating. This seems to take two forms:

1. They don't write anything down and forget what they've learned
2. They don't know how to chat with each other

The first one can be solved with the earlier point about checklists. If you get people into the habit of writing stuff down all of the time, the wiki no longer feels like a daunting place where text goes to die. Instead, it becomes a thriving, living, breathing document that everyone contributes to. But it needs to become habit before it can really add value.

The second point is really tough. If you've got thorny personalities or people who just don't know how to "place nice" over Slack can come off as curt and abrasive even if they don't mean to. I know plenty of folks in software who want to use as few words (and punctuation) as possible to get their point across. I get it - time is of the essence for a software engineer.

The problem is that when you make the answer to everything

> k

you make people really uncomfortable. Like it or not, people assume the worst and will analyze the crap out of a vague message. Was it dismissive? Does "k" actually mean acknowledged or go away?

To fix this, you probably need a page in your wiki on something to the effect of [Slack etiquette](https://slackhq.com/etiquette-tips-in-slack). For some, this may seem ridiculous that in this day in age we still need to train adults on how to play nicely with each other.

But we have to eat our own dog food here; you have to assume positive intent yourself that maybe some people just don't realize they look mean and nasty even if they don't intend to and they genuinely have never had issues before communicating in such a Spartan fashion. If that's the case, your etiquette guide serves as a friendly primer.

## All is not lost

If we look at all of these habits above, they apply to high-functioning team and can be used to transform dysfunctional teams. **It just takes work.** You need to put the effort in to transform an under-performing team. You also have to have the patience to see it through. Not everyone has the stomach to [turn the ship around](https://www.youtube.com/watch?v=ivwKQqf4ixA) but if you're in this position there is hope for you yet!
]]></content:encoded>
      <dc:date>2020-04-21T14:09:19+00:00</dc:date>
    </item>
    <item>
      <title>Working Remotely When You Didn't Plan To</title>
      <link>https://www.adamconrad.dev/blog/working-remotely-when-you-didnt-plan-to/</link>
      <description><![CDATA[If you’re reading this article at the time of its publishing, your professional world is probably upended with pleas (or demands) to work from home. I’ve embraced remote work culture for years and I believe it is the future of work.

]]></description>
      <pubDate>Thu, 27 Feb 2020 23:00:19 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/working-remotely-when-you-didnt-plan-to/</guid>
      <content:encoded><![CDATA[If you're reading this article at the time of its publishing, your professional world is probably upended with pleas (or demands) to work from home. I've embraced remote work culture for years and I believe it is the future of work.

Companies that embrace remote-first can increase the size of their candidate pool, reduce costs in lower cost of living areas, increase employee satisfaction by removing one of the worst things about a job (a commute), and hold a key advantage in flexibility that their competitors cannot leverage.

Because of this, my teams at [Indigo](https://indigoag.com) are remote-first and the majority of my team members live outside of the Boston metro area where our headquarters resides. Not only are we ready and able for remote work, we also perform at a very high level within our organization.

How are we able to do this? I'll share **three key strategies that are essential for remote work to succeed in today's workplace**.

## Over communicate

This may sound obvious to certain leaders but this cannot be understated: **digital communication requires overcommunication**. There are a few reasons you need to overcommunicate when moving to a remote organization:

1. Words can be misinterpreted. One's voice does not only communicate words, but tones, inflections, and emotion. This is all removed when your default mode is text rather than speech. Reinforcing this with multiple takes (and multiple mediums) for the same message will ensure no one gets the wrong message.
2. Text can feel restricting. As a rule of thumb, if you don't feel like your message is getting across with 3 attempts, **get on a video call or phone call with the other people.** Just because you are remote doesn't mean you can't fall back on video or audio to emulate face-to-face communication.
3. Not everyone will see your message when you send it. With remote comes an increased risk for asynchronous messaging and reduced priority that your message will be heard. The more times you say the same message, the higher the likelihood that everyone will get what you're saying when you say it.

So **say the same thing over, and over, and over, and over, and over again until you are 100% confident that everyone gets it**. They may not have seen it the first time, or interpreted the way you did, or could use the help of a personal chat with a tool like Zoom or Slack.

## Document everything

As I mentioned in the previous section, your medium has shifted from real-time, synchronous communication, to asynchronous communication. This is both a blessing and a curse.

It is a blessing because you will are now in control of your communication channels. In the office, if someone comes up to your desk and interrupts you, you have no agency as to whether or not this message is worth your time. Some people want to come by and chat. Others need your undivided attention. Now if eight people bug you on Slack, _you_ get to decide when and how to message them back. This is very freeing and allows you to take better control of your time.

It is also a curse because you now need to be on top of the myriad communication channels that exist. Some people love email, others love Slack, and still, others will use text message or a phone call to reach you. Guess what? You're now responsible to check all of them which will only continue to fraction your time with distractions.

To combat these problems, you need to **document any notes, action items, or checklists that could be useful to more than one person**. When you shift your mindset towards asynchronous communication, you are using documentation as your key lever. **Documentation is your leverage so you don't have to repeat yourself**. When you don't have to repeat yourself, your time doesn't have to be fractured answering so many questions on Slack or email.

How to get started with documentation is easy: write stuff down! Hopefully, in a centralized place. Unfortunately, in some organizations, there are many different tools used at once. For example, we use Microsoft OneDrive for Office _and_ Google Drive _and_ Confluence _and_ an internal Facebook wiki.

Yikes.

If possible, try to converge on one shared wiki tool and stick with it. For engineering, we use Confluence within the Atlassian ecosystem. So you know that if I wrote an article, it's on Confluence. **Do your best to write your content on one platform and make it known to help your coworkers reinforce the need for one central system.**

## Trust & transparency

The bedrock that supports all of the above is trust. Another one that seems obvious to many leaders but creeps insidiously on newly-remote organizations. Why? **When you go remote, you lose control over your team's time.**

This may be a hard one for you to hear but if you're used to showing up in the office you're likely preconditioned to assume that "butts in seats" equates to productivity. How will I know if my team is doing work and not slacking off? Are they putting in a full effort if they have a TV and a bed close to their desk?

I hate to break it to you, but you need to `l e t   g o`. That's right, we've got a control complex. And I get it; the American employment system since the Ford days has conditioned us to believe that productivity is correlated with in-office work.

Let me offer you a quick analogy as to how this changed my whole approach to trust with my organization:

### A quick aside on trust over a lifetime

When we were children, the world gave us structure and guidance because we were not capable of making sound decisions for ourselves. School starts at a specific time, you must eat the lunch given to you, and your whole world is regimented to the whims of the adults you're surrounded by.

Once you got to college, that whole paradigm changed. Professors did not care if you did the homework, showed up to class, ate lunch, or really did anything. If you don't do the work, you simply fail. In that sense, all you have to blame is yourself.

And then something curious happened.

You got a job, congratulations! You've now regressed to being treated like a child again: you must show up to work at a specific time, you have a narrow window for lunch, and your boss gives you dirty looks if you leave at 4:58 PM to pick up your kids.

Does it have to be this way?

### Trust means assuming positive intent

The truth is that we can reverse this trend of no trust. **If you assume positive intent, your people will being to trust you and be more compelled to deliver great work.** When you _decide_ that your people are great (because you only hire great people) from the beginning and don't need to be watched over by you to do their best, you are assuming positive intent.

This new world of remote work may mean they take a longer lunch break. Or a break to feed the baby. Or go to the gym. So what? At the end of the day, if they do the work, who cares if they're online at 3 pm or 3 am?

It's certainly helpful to your teammates to know others are online to assist them. It is preferred everyone is online at the same time, but it is not required for success.

**With trust comes accountability.** Now that you are assuming positive intent and trust in your teammates, they need to return the favor by holding themselves accountable. When the boss isn't looking, how do _I_ prove to the world that I'm delivering on my work? How can _I_ demonstrate that I'm worthy of a promotion or raise?

If you've got anxieties like this moving to remote work, the focus has shifted from time to results. _And that's a good thing_. When we all trust each other to get things done, our value is solely reflected on our output and not on inconsequential metrics like attendance or time on the clock. It's liberating; let remote work liberate you!

## We've got a tough road ahead, but we can get through it together

If you keep these three tactics in mind you'll be shocked at how much your teams will begin to embrace remote work culture. **Be sure to balance all of this work communication with social communication**. That means creating fun Slack channels like `#music`, `#video-games`, or `#cute-animals`.

With everyone holed away in their home offices things are going to be quite isolating. As humans, we crave social interaction. Recognize this and don't make it all about business. Lighten up and do your best to support your teammates. We're all going to make it and having a sense of humor will help us all get through this together.

Best of luck!
]]></content:encoded>
      <dc:date>2020-02-27T23:00:19+00:00</dc:date>
    </item>
    <item>
      <title>Technical Lead Management 101 or How to Try Out Management</title>
      <link>https://www.adamconrad.dev/blog/technical-lead-management/</link>
      <description><![CDATA[
  I wrote this for my tech leads who needed a crash course in engineering management. The advice you see in this article is the advice I used to help level up my team leads.


]]></description>
      <pubDate>Thu, 27 Feb 2020 23:00:19 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/technical-lead-management/</guid>
      <content:encoded><![CDATA[> I wrote this for my tech leads who needed a crash course in engineering management. The advice you see in this article is the advice I used to help level up my team leads.

At some point in your technical career you'll have to make a choice: stay on the individual contributor (IC) track or move into management. Many seem to believe that the latter is the only way to move up in your career.

It's not!

Plenty of people want to (and deserve) to grow their careers as software developers without having to manage others. **Regardless, at some point you'll have to test the waters to see which is right for you.** To ease this transition and minimize the risk of inundating developers with too much change, we've created a hybrid role called _Technical Lead Manager_. In this article, we're going to describe what this role is, why it is needed, and the basics for becoming a technical lead with management responsibilities.

## What is a Technical Lead Manager?

While there is no official definition of this role, in my mind **a Technical Lead Manager (TLM) is a tech lead with minimal engineering management responsibilities.** More specifically, we define "minimal" to mean **no more than 3 direct reports.** If you're brand new to engineering management, a _direct report_ means someone (e.g. a software engineer) who reports to you in your company's organizational hierarchy. In other words, you write code _and_ are the boss of up to 3 people.

### Why not call them a Junior/Associate Engineering Manager?

You could, but the distinction here is that **we expect an engineer with this role to still perform his/her technical duties (i.e. coding, debugging, code review), albeit in a more limited capacity.** Without the _Technical_ adjective, we leave the opportunity to let those technical skills atrophy in certain organizations.

### Why the need to combine both duties? Won't that overwhelm the individual?

At some point in your career, you'll be faced with the decision of considering roles in management. In many organizations, the leap to engineering manager is a large one. Make no mistake: **engineering management is a completely different skillset from software engineering and should be considered hitting the "reset button" on your skills.** With such a stark contrast in roles and responsibilities, the transition to management at the most junior levels could dissuade people from considering this career move in the first place.

Nonetheless, leaders need to groom the next generation of leaders underneath them. To ease the transition and best prepare their future leaders with the necessary skills, consider looking at this hybrid role.

## What does a Technical Lead Manager do?

At its core, a Technical Lead Manager codes _and_ manages the career of up to 3 direct reports. Since management is generally the newer of the two skills, we won't focus too much on how to develop the technical skills. For further reading on this topic, consider [this primer](https://www.getclockwise.com/blog/the-new-tech-leads-survival-guide) if you've never been a tech lead before.

Instead, let's dive into the core skills you're going to have to build from scratch.

### The Four Dials of Engineering Management

When I first became an engineering manager, my CTO gave me this analogy to think of engineering management skills _like knobs on a guitar amp_. For EMs, there are four dials: **people, process, product, and prowess**.

Each dial is turned up somewhere between 0 and 10 depending on the organization you work in. Some companies highly value technical prowess, while other companies see EMs as people managers with a previous technical background. Regardless of your company's organizational design, you're going to want to know what all four knobs are so you can dial in the exact volume of each skill.

#### People

Let me cut right to the chase: **successful engineering management is all about maximizing the effectiveness of your people.** If you take nothing else away from this guide, know that _simply caring about your people will get you 80% of the way there_. To effectively care for your people, this dial itself has 4 core aspects:

##### One-on-ones (O3s)

These are the bread and butter of effective management. **Consider them the most sacred time of your entire week.** O3s build trust, shared understanding, and rapport. From all of the materials I've read on this subject, here's a formula for the most effective O3s:

* Have them every week with all of your directs for 30 minutes.
* If you have a conflict, move your O3 but _never cancel it_.
* Start each O3 with "how are you?" or "what's on your mind?"
* At first, allot all 30 minutes for your directs to say whatever is on their agenda. They may have a lot on their mind in getting to know you so indulge them.
* At some point (around a month in), you'll want to shift it to be 15 minutes for their agenda followed by 15 minutes for your agenda.
* Finally, another month or so in, you'll want to trim a few minutes off of your agenda to bake in coaching & goal setting for your directs' career development.

I still use this formula for all of my O3s and it has served me very well. **Be an [active listener](https://www.youtube.com/watch?v=saXfavo1OQo) during their agenda.** If you're looking for what to ask during your agenda, I've referred to [this list](https://larahogan.me/blog/first-one-on-one-questions/) for seminal O3 questions to ask your direct reports. For coaching and goal setting, keep scrolling. But first, you're going to want to make feedback a central tenant of your agenda.

##### Feedback

People crave feedback. **Being an effective manager means being an astute observer.** There are lots of ways of observing your team without conjuring up an image of the nosey micromanager. You're  probably doing a few of these already:

* Active listening in meetings
* Addressing blockers in stand-ups
* Reviewing code
* Pair programming

Plus countless more. The difference now is you're not just observing with the lens of a software engineer but also as a manager with a chance to provide feedback.

**Feedback is about behavior.** Let me repeat myself: feedback is _not_ about changing people but about changing behaviors. "You're bad at your job" is not feedback. It is an ad hominem attack.

Feedback is either about _encouraging a behavior (positive)_ or _stopping a behavior (constructive)_. Feedback, too, has a formula for being given effectively. The one I like to use is based on [nonviolent communication](https://en.wikipedia.org/wiki/Nonviolent_Communication). The formula, like seemingly all of my formulas, occurs in four steps:

1. **Describe the current behavior.** Example: "When you test your code..."
2. **Describe the outcome of that behavior.** Example: "...it increases the reliability of our product."
3. **Describe the effect that behavior has on you or others.** Example: "This instills trust in our customers that we build software that works."
4. **If the feedback is positive, use encouraging language to reinforce the behavior. If the feedback is constructive, describe how you would like the behavior changed in the future.** Example: "Keep it up!"

Stitching this together, our positive feedback example looks like this:

> When you test your code, it increases the reliability of our code. This instills trust in our customers that we build software that works. Keep it up!

Conversely, an example of constructive feedback would look very similar, with the largest difference being the final fourth step:

> When you are late to meetings, it reduces the time we have to get through all of our agenda items. This inconveniences the team and reduces their trust in you to be reliable and timely. Please be on time from now on, thanks!

Study this last example. There are a few things at play here that you should pay attention to:

* The format for both positive and constructive feedback is the same. This conditions your team to see all feedback as a focus on behavior and not a personal attack on who they fundamentally are as human beings. If you keep all feedback formats the same, they won't cower at the thought of receiving negative feedback, and they won't have to worry about whether you're going to praise them or bring them down.
* For constructive feedback, it's about the future, not the past. We can't change the past, so there is no reason to dwell on it. Instead, accept that the undesired behavior happened; don't dwell on it or berate your teammates for their shortcomings. All we can do is stop undesired behavior from happening again by steering the rudder in the right direction.

If you follow these guidelines, your team will think of all feedback, positive or constructive, as a window into effective reinforcement or behavior change.

**Finally, I would recommend you wait to introduce feedback until you've had about 8 O3s (and start with only positive feedback).** You need enough time to build rapport with your directs before you begin to give them feedback. And even then, you want it to start positive. Add on another 8 weeks (16 in total) before introducing constructive feedback so they are used to your communication style.

##### Coaching

Once you are having regular meetings with your directs and giving them regular feedback, it's time to start coaching them in their careers. Coaching can take many forms, some of which I've already mentioned before:

* Pair programming
* Code review
* Mentoring
* Career development plans

I use all of these tools regularly and you should feel free to use all of them. **The point of coaching is to get more out of your directs.**

O3s build trust. Feedback is the transparency that results from that trust. Once you have transparency, you now know what your directs are capable of. To be a next-level manager, you need to strive to extract more value out of your directs by pushing them to be their best selves.

How do you get more out of your directs? **Reinforce their strengths and bring up their weaknesses.**

Do they write amazing tests? Have them review others' code for sufficient test coverage or run a workshop on effective testing strategies.

Conversely, if they constantly forget to test their code, consider introducing a pull request checklist to remind them to include tests and enforce that all code comes with tests to merge code into production.

You're going to have to get creative at this step because there are [boundless ways to coach someone](https://knowyourteam.com/blog/2020/02/19/how-to-coach-employees-ask-these-1-on-1-meeting-questions/) through their career. With all of this new work to do, how are you going to find time to still code?

##### Delegation

The only way to find time for everything is to lean on your directs. **Delegation is the skill of pushing down responsibility to others.** I like to think of this with an analogy:

Imagine the CEO of your company. He or she is likely the only person in the company with the knowledge of the entire ecosystem of how your company operates. They're also likely the only person who can both sell the product to investors and create a grand vision of how your company can wow your customers. The value of these two skills let's say is $500/hour.

Now imagine a paralegal at your company. Paralegals often review documents, perform research, and write legal papers on behalf of the lawyers they work with. Let's say these skills are valued at $35/hour.

Your CEO may have a law background and is quite capable of performing the duties of the paralegal. But if the CEO can do the legal work _and_ raise money from investors, while the paralegal can only do the legal work, why would you have the CEO ever do legal work? It's in the best interest of the company for the CEO to focus solely on the work he/she is uniquely suited to perform.

**In your case, you are uniquely suited to manage and tech lead. If you are doing _anything_ outside of those two things, you are reducing the effectiveness of your position and should delegate that responsibility to others.**

Delegation is the hardest part about being a TLM (or any manager for that matter). You are naturally going to want to do what you're comfortable doing: coding. Fortunately for you, coding is no longer your highest value skill.

**At every possible moment, you should evaluate whether the task you are performing could be done by someone else.** If it easily can, document the work involved and share it with your team to perform. If it cannot, then it's probably something you need to continue to do yourself. This self-evaluation is the basis for the next dial of engineering management: process.

#### Process

If engineering management centers around people, then Process is the mechanism for making the People dial as efficient as possible.

**Process is all about documentation.** If you have a repeatable, verifiable way of doing something, you should document it for your team or others to perform. Fun fact: this post is a manifestation of my process. Teaching others how to be engineering managers is a process. Reading this post is the process by which engineering managers can learn about how to manage for the first time. I know this is all getting pretty meta, so let's focus on a few ways to institute process:

##### Leverage a wiki tool

Tools like Confluence or GitHub Wikis are a great way to codify processes. The question I always ask myself is:

> Do other people need to know this and will forget it? I should write this down in the wiki.

I mean all of the time. Even the most seemingly mundane tasks are documented in our Confluence at work. **Never assume a task or process is too dumb to spell out in your documentation tool.** As we gain experience in the field we assume a lot of things about our work. Not everyone has the same experiences or background as us, so what may seem trivial to you could be quite complex to others.

That's why it's always a safe bet to write it down in the docs.

At worst, you took some time to become a better writer and communicator. At best, your documentation with the whole company and adds value to everyone around you. So why wouldn't you write it down?

##### Use it as a teaching moment

Looking for ways to incorporate coaching into your workday? Describe a process to a peer or direct. Gather feedback on your process: what works? What doesn't? What could be improved? Processes should help make everyone around you more effective. If you can teach something to others, you're building your skills as a coach and adding value to the organization.

##### Simplify your life

The beauty of processes is once it is set in stone, you pretty much never need to think about it again. Processes turn algorithmically intense into the automatic.

For example, if you didn't have a process for running effective meetings, you probably spent a lot of cognitive effort in figuring out how to set the agenda. Once you have a process, though, planning meetings become quite rudimentary and painless. You have a plan for how you're going to get through them.

Processes make your life easier, and your life is already challenging enough.

#### Product

One of the cool parts of moving into management is the cross-disciplinary skills you'll develop. **Get ready to become best friends with your product manager.**

I naturally consider myself to be a product-minded engineer. If this doesn't come naturally to you, it will be thrust upon you whether you like it or not. As the engineering manager, **you are now the liaison to other departments about the technical capabilities of your team.** Think of yourself as the head consultant and advisor to the rest of the company about your team.

What is the makeup of the skills on your team: are they front-end or back-end focused (or full-stack)? Who loves tackling bug tickets and who loves fixing tech debt? How will you balance tech debt and bugs with new features? Who decides whether a bug is more important than a feature? These and countless other questions will arise when you turn up the Product dial.

**Your best defense in dialing up Product is to be on the offensive, and that means getting to know your product and your product managers.** Just like your directs, set up O3s with your product manager. Get to know them and what makes them tick. Build trust. The only way you can build great software without you and your PM at each other's throats is to create an authentic relationship.

**There will usually be a healthy tension between product and engineering and that's okay.** Sometime you'll want to address mounting tech debt in your end-to-end tests. Meanwhile, your PM will wonder why Widget X wasn't delivered yesterday. You will each have reasonable justifications for why your stance is correct. Guess what? At the end of the day, you're going to have to come to a consensus. **See my earlier points about active listening, nonviolent communication, and building trust.**

Oh, and by the way, this extends far beyond Product. You'll have conflicts to resolve with architecture, design, marketing, sales, legal, and virtually anyone who comes in contact with your team: _including the team itself_. One engineer will say that Vim is the best IDE. Another will say it's Emacs. Spaces vs tabs. Linux vs Mac. It's pronounced "gif" (no it's "jif"). You name it, someone will have an opposing opinion.

**You are mediator first, moderator second, and arbiter last.**

What I mean by this is in any conflict, your first job is to _help each side empathize with the other_. Empathy, often confused with sympathy (which is about feeling someone else's distress), is about walking a mile in another person's shoes. Can you get Person A to see where Person B is coming from, and vice versa? If not, move into the moderation phase.

Moderation is about guiding the discussion to help the two parties address their concerns and air their grievances. This guidance should hopefully lead to automatic mediation: by moderating the discussion, you're allowing both parties to see where the other is coming from in a way you couldn't immediately achieve in your first pass. But what if consensus should not be reached?

As a last resort, it's time to be the judge, the jury, and the executioner. Ideally, people come to their conclusions themselves. The advice I've always latched onto is

> Your best ideas seem like they never came from you at all and instead came from the person you were trying to convince in the first place

Sometimes you don't have that luxury. In that case, you have to make a call for others. An often-quoted maxim of the military is **a poor decision is generally better than indecision.** Dwelling on whether you are making the right call in this argument and coming up with nothing is worse than just picking one side.

This Product dial morphed into conflict resolution but is an important aspect of the natural tension that will arise when you are no longer interacting solely within the discipline of engineering.

#### Prowess

Finally, we can't forget what got you here in the first place: your technical prowess. For TLMs, coding, tech leadership, and review should still be anywhere from 50 to 80% of your duties depending on the organization you are a part of. In my first formal engineering management role, I also played the role of a tech lead and spent nearly 80% of my time coding. The further up the food chain you go, the less you will be coding. **Do whatever you can to keep that number above 0%, even if that means coding after work.**

Why? **No one wants to work for someone who can't remember what it's like to be in the trenches.** The engineering managers I've respected the most are those who can still tango with the code. Like the CEO and paralegal analogy from above, you _could_ code, but you're choosing to delegate some of those tasks because a portion of your time is better spent managing.

I find that my technical prowess is best utilized through code review and coaching. Writing out a folder structure for React components is suboptimal compared to teaching a room full of engineers on how to organize their React projects.

**You only have two hands. Every teaching opportunity is a chance for you to give yourself more hands.**

**Systems Design becomes the primary skill** you'll be building in your technical dial. Now that you can't spend all of your time programming, you need to be thinking at a higher level as to how your systems will connect to operate in an efficient, scalable, and reliable manner.

[I love this Systems Design playlist on YouTube](https://www.youtube.com/watch?v=quLrc3PbuIw&list=PLMCXHnjXnTnvo6alSjVkgxV-VH6EPyvoX). Not only is it a good source of Systems Design interviews to implement in your hiring program, but it will build your skills when you inevitably interview somewhere else and want to demonstrate you know how to logically think about systems. That playlist will help you get the reps in and build this skill quickly. Do yourself a favor and watch all of the videos on this list.

### Other duties of the TLM you should know about

If you stick to mastering the Four Dials of Engineering Management and maintain your skills as a tech lead, you'll find the Technical Lead Manager role to be very rewarding and easy to transition into. Depending on your companies needs, there are a few other duties I want to call out that you should be aware of that EMs also do so you aren't blindsided:

* **Hiring.** When the money is rolling in, you're inevitably going to be growing your team. You're not only going to be on the other side of the interview, but you may not be interviewing for just technical skills. Oftentimes engineering managers assess things like culture fit and soft skills. I'll save this one for another post, but for now, I would just Google how to run a cultural interview because there are plenty of dos and don'ts you need to consider to keep your interview above board.
* **Firing.** With great power comes great responsibility. As much fun as it is to bring in great people, it is equally crushing to let go of underperformers. Performance management is all about retaining great talent and coaching your direct reports to be the best they can be. Sometimes that's just not enough. The key here is to **document everything and ensure you have a strong case for why this person cannot be saved.** This cannot be a gut feeling. You need plenty of documentation of continued underperformance and failure to adopt feedback before you should consider things like PIPs (Performance Improvement Plans). When in doubt, check with your HR or People team on how to handle these scenarios.
* **Performance Reviews, Compensation, and Promotion.** If you have great people and you are following the Four Dials of Engineering Management to get the most out of your team, you're putting them on the path to upskilling them. They will recognize their metamorphosis and demand more pay or a title bump. Like the previous point, these merits cannot be based on gut feelings alone. Instead, _help your directs build an ironclad case for why they should be recognized_. Someone will ask you to justify why your direct deserves a promotion and a bonus. If you can delegate the rationale to your direct, not only will they be able to justify their raise, but so will you.

## Conclusion and Further Reading

This post is long enough (and I'm tired enough) that you should be well on your way to a successful trial at engineering management. I want to thank [this post](https://randsinrepose.com/archives/meritocracy-trailing-indicator/) for inspiring me to consider Technical Lead Managers as a great bridge to delegating management responsibilities to your team.

All of these tools and strategies were tried and tested by me over the last several years as an engineering manager. All of them are backed up by the amazing engineering management books, podcasts, and blogs I've read over the years. An exhaustive reading/listening list can be found [here](https://leadership-library.dev), though I'd like to call out a few pieces I've found especially useful over the years:

* _The Effective Manager_ by Mark Horstman (and the accompanying _Manager Tools_ podcast)
* _Resilient Management_ by Lara Hogan
* _High Output Management_ by Andy Grove
* _The 27 Challenges Managers Face_ by Bruce Tulgan (particularly for conflict resolution)
* _Accelerate_ by Nicole Forsgren, Jez Humble, and Gene Kim
* _An Elegant Puzzle_ by Will Larson
* _Extreme Ownership_ by Jocko Willink (and his _Jocko Podcast_)
* _The Making of a Manager_ by Julie Zhuo
* _The Manager's Path_ by Camille Fournier
* _The Talent Code_ by Daniel Coyle

I've read all of these books and are the basis for how this guide was built. If you want to learn more about anything discussed in this guide, I would encourage you to read the source material above. Let me know what you found most helpful or if you'd like clarity on any of this [on Twitter](https://twitter.com/theadamconrad) and I'd be happy to walk you through any of the finer points.

Best of luck and remember that **management is a skill just like anything else**. With a combination of study and practice, you'll be well on your way to greatness!
]]></content:encoded>
      <dc:date>2020-02-27T23:00:19+00:00</dc:date>
    </item>
    <item>
      <title>The teamwork analogy I use nearly every week</title>
      <link>https://www.adamconrad.dev/blog/the-teamwork-analogy-i-use-nearly-every-week/</link>
      <description><![CDATA[I remember when I started at my current company there was a departing employee who had a reputation for being exceptionally talented but also quite difficult to work with. This was so pervasive that virtually every person I spoke with had the same thing to say about them. It became very clear when they left how much of a hole they left because of their prowess.

]]></description>
      <pubDate>Sun, 12 Jan 2020 22:06:12 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/the-teamwork-analogy-i-use-nearly-every-week/</guid>
      <content:encoded><![CDATA[I remember when I started at my current company there was a departing employee who had a reputation for being exceptionally talented but also quite difficult to work with. This was so pervasive that virtually every person I spoke with had the same thing to say about them. It became very clear when they left how much of a hole they left because of their prowess.

The code was smart and quite clever. In fact, it was so clever that it almost impossible to understand; to the point where we ended up scrapping all of it in favor of something more verbose but much simpler.

Was it by-the-book the best code we could have written? No. But it was code that everyone on the team could understand and contribute to, which made it way more versatile.

I think back to these first few weeks (and this old code) quite a bit in my one-on-ones and onboarding meetings. It made me think of an analogy I find myself using over and over again to show people why teamwork is so important. It goes something like this:

> No one ever partially IPO'd. The NYSE isn't going to have you ring the bell because you're a rock star programmer while the rest of the company fails. Either we all cross the finish line, arm-in-arm, or we all fail to complete the race.

So many people fail to see their petty arguments with their fellow employees are completely fruitless. What is the point of winning your argument if it fractures your team?

Companies are more like baseball than tennis; it's not an individual sport. One or two stellar performers cannot possibly make up for a losing team. That's why I don't hire divas or prima donnas.

Sure, the rock star may be exceptionally talented and even a "10x" programmer. But if their 10x causes everyone else to drop 50% in productivity because their code is exceptionally clever or they are overly critical in code review, then the team overall is still no better off in terms of productivity.

**If you want to be selfish, it's in your best interest to be a team player.** You read that right, if you want to make that sweet IPO money the only way you are going to get there is if your whole company gets there. It's in your best interest, therefore, to help others out so you can all get to that exit event sooner rather than later. You only get there after the last person crosses the finish line, and you're all linked arm-in-arm. _That's_ why you stand to benefit if others benefit.

The next time you're ready to rip someone's throat out for a bad move or you think your marketing team is useless, remember that every piece of the puzzle has to fit if you want to win. Otherwise you'll stay stuck in the same rut and no one wants to work with someone who is difficult and not a team player.
]]></content:encoded>
      <dc:date>2020-01-12T22:06:12+00:00</dc:date>
    </item>
    <item>
      <title>Always have the Beginner Mindset</title>
      <link>https://www.adamconrad.dev/blog/always-have-a-beginners-mindset/</link>
      <description><![CDATA[The Beginner’s Mindset is about approaching learning with open arms and the understanding that you don’t know everything. You intuitively have this approach when you actually are a beginner but if you maintain that approach as an expert it will serve you well.

]]></description>
      <pubDate>Fri, 13 Dec 2019 12:19:06 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/always-have-a-beginners-mindset/</guid>
      <content:encoded><![CDATA[The Beginner's Mindset is about approaching learning with open arms and the understanding that you don't know everything. You intuitively have this approach when you actually are a beginner but if you maintain that approach as an expert it will serve you well.

I still keep this mindset with everything I do because staying humble and thirsty to learn keeps you ahead of your peers and you'll learn faster. Here are a few things I keep in mind when learning with a beginner mindset.

## Learn how to learn

It sounds like a ridiculous premise given that many folks spend much of their childhood in a classroom learning. Unfortunately, most schools teach but that doesn't mean students learn. And like anything, learning is a skill that can be acquired (and is not taught in primary school).

The best way to learn this is to [take this Coursera course](https://www.coursera.org/learn/learning-how-to-learn/) which is really both a class on how to learn and how to read effectively. And while you're at it, you might as well learn how to [speak](https://www.youtube.com/watch?v=Unzc731iCUY), [write](https://www.youtube.com/watch?v=vtIzMaLkCaM), and how to [think critically](https://www.coursera.org/learn/mindware?).

## Throw out your assumptions

If you're starting to practice the beginner's mindset a decade into your career, you'll have a lot of knowledge already built up. This will only bias you into thinking like an expert, which will hinder your progress.

Instead, throw out all of your pre-existing assumptions and **assume you might be wrong**. This opens you up to new possibilities that you likely didn't consider because of all of your inherit biases you've built up over the years. Now you can explore ideas and concepts with a whole new chain of connections that you'd previously cut off.

## Stop jumping to solutions

Another problem with thinking with your experience is when you see a new problem, you rely on your previous solutions to create a novel way to solve this one. Again, if you think like a beginner, you don't have previous solutions to rely on. What you think is a perfect solution is no longer relevant so you should scrap it.

The beginner's mindset means **asking why all of the time**. This is so effective there is an entire [Wikipedia entry on this concept](https://en.wikipedia.org/wiki/5_Whys) that I would encourage you to investigate further.

## Beginners dream

Something happens as we gain experience: we also gain cynicism. We see what has succeeded and what has failed and that hurts our ability to think ideal states. We know the world is shades of gray but why should that hinder us from believing beyond our true potential?

My favorite example of this is Airbnb's [11-star customer rating framework](https://uxdesign.cc/applying-airbnbs-11-star-framework-to-the-candidate-experience-3f0b9c4e68a3?gi=88688bfe0579). Everywhere you go you either see a thumbs-up/thumbs-down rating or a 5-star rating system. Airbnb wanted to know what happens if you break the glass ceiling.

What would a 6-star rating on Airbnb look like? I mean, what does more than 100% look like to you? That must be pretty great.

Well, then what does a 7-star rating look like? Can I even fathom that kind of an amazing experience?

Okay then tell me about a _10-star experience_. Uuh...what are we, the Four Seasons hotel in my tiny Boston apartment? Hey! That's interesting; are there any elements of the famous Four Seasons hotel chain that we can replicate even in the smallest apartments we list on Airbnb?

Beginners don't know where to stop because they've never reached the end before. That means they are free to dream the greatest of dreams without limitations. This unshackling of limitations is not only liberating, but idea-generating. What would an 11-star experience look like for _your_ business?

## Embrace the haters

Another issue with cynicism is you know where you might fail. You anticipate criticism and faults so you are quick to deflect criticism with retorts rather than _embrace_ it.

What if that criticism is advice? What if, what sounds like criticism, is actually a cry for help? And finally, what if the critics are right and you are wrong?

When you think like an expert, you assume you are right from the start. When you are a beginner, you assume you are _wrong_ from the start. People who want to learn and grow will crave feedback and help if they think they don't have all the answers.

Can you see how limiting it might be to think like an expert?

## Teach what you learn to internalize

Finally, just because you learned something doesn't mean you know it. How many TED talks have you listened to that went in one ear and out the other? If you took the above course on how to learn and you're checking back in here, you should know a great way to internalize what you've learned is to revisit the material and ideally write about it. And what do you know, a platform like a blog is a perfect way to teach others and engage your mind with active learning techniques.

Trust me, nothing is more gut-wrenching than trying to write a blog post on a technical topic and posting it to the internet. The best defense against looking stupid is to really know the material. Writing is a great incentive to force yourself to learn what you are talking about so you aren't embarrassed.

The beginners mindset has allowed me to stay humble, stay open to possibilities, and most importantly, stay hungry to learn and grow as I move into the next phase of my career. Hopefully it will do the same for you as well.
]]></content:encoded>
      <dc:date>2019-12-13T12:19:06+00:00</dc:date>
    </item>
    <item>
      <title>Running Effective Planning Meetings</title>
      <link>https://www.adamconrad.dev/blog/running-effective-planning-meetings/</link>
      <description><![CDATA[As a technical lead, planning meetings serve as your opportunity to clarify upcoming work with your stakeholders. Running them properly will make your meetings more effective both for your teammates and your stakeholders.

]]></description>
      <pubDate>Mon, 04 Nov 2019 13:17:19 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/running-effective-planning-meetings/</guid>
      <content:encoded><![CDATA[**As a technical lead, planning meetings serve as your opportunity to clarify upcoming work with your stakeholders. Running them properly will make your meetings more effective both for your teammates and your stakeholders.**

In software, planning meetings serve two primary missions:

1. Clarify what your team needs to do.
2. Give stakeholders confidence in the work you are promising to perform.

How do you approach these meetings to achieve these goals? Follow these steps:

## 1. Revisit work in progress from last week

The first step is to review the work that has carried over from last week. Go through every task with each developer and ask the following questions:

* We didn’t get it done last week - will we get it done this week?
* If so: are the acceptance criteria still clear?
* If not: do we need to break up the ticket? Are there blockers? **What has changed since last week?**

If we didn’t land our commitments then we need to quickly learn **why we missed our commitments and what needs to be done to fix it.** It could be that the requirements were too vague and need clarity. It could be that the requirements were clear, but to implement the functionality, we surfaced additional concerns that will result in additional subtasks and work for the team. **It is essential to follow through on existing work or deprioritize this work before we add new tasks in from the backlog.**

## 2. Pulse check, the first: are we full for the week?

It is possible (though uncommon) that so much is still in progress that it is not wise to add in more tickets. **If you overcommitted last week then the key is to nail those commitments this week.** Software is difficult to estimate; you won’t be perfect and that isn’t the point. The point is to **manage expectations with stakeholders and strive for better estimation every week.**

## 3. Search the backlog for additional work

If your team isn’t fully utilized for the upcoming week then it is time to look into the backlog to find more work for your team to do. How do you know what work should be performed next? Depending on your team, it should look like one of the following scenarios:

1. **What’s at the top of the backlog?** If you have groomed your backlog with a Product Manager, then the backlog is likely prioritized from top to bottom.
2. **What is the highest priority item?** Priority is another ranking mechanism. Perhaps an IT support ticket came in at the last minute, so it is at the bottom of your backlog but surfaces an important production issue. This likely needs more attention than tickets even at the top of your backlog.
3. **What are your team objectives? Is the order or priority not in alignment with certain near-term objectives you’re trying to accomplish?** Are you launching an MVP? Do you have a checklist of things that must launch before the end of the quarter? Quickly scan the backlog for other tasks or stories that may have missed the first two scenarios.

Once you’ve found the work, it’s time to move onto the next step.

## 4. Clarify the work for each task you add from the backlog

Now that you’ve determined there is work to be done, here’s a sample checklist to ensure the work is ready to be performed by your development team:

* Does the title correctly summarize the task to be completed? (If it’s a story, does it follow the format "_As a PERSONA, I want to PERFORM ACTION so that I can ACCOMPLISH GOAL_")
* Is the description filled out with acceptance criteria? What does the Product team/Customer expect will be available when this ticket is complete?
* If this is a story, what the subtasks for engineers to complete to assert the story is done? What is the “definition of done” for this ticket?
* What team will be working on this? API or UI? A product team or a platform team? Is this even on the right Jira board?
* Were any comments or updates left to indicate more information that wasn’t captured?
* If this is a UI task: are there mockups or InVision prototypes? Are those mockups linked appropriately?
* Speaking of linking: is this ticket blocked by another ticket? Follows the completion of another ticket? Is there any kind of connection from this ticket to another?

This is by no means an exhaustive list. This is simply some of the things you should consider when you’re evaluating if **the ticket has all of the information necessary for _any developer on your team to complete the ticket_.**

## 5. Add the ticket to this week’s scope & repeat step 4.

Not much more to say here other than to repeat step 4 until you reach step 6…

## 6. Pulse check, the second: are we full now?

Steps 4 & 5 can get unwieldy if you’re trying to eat clock in the rest of your meeting. Next thing you know you’ve groomed quite a few tickets that may or may not still fit into the work capacity of a given week. So let’s check again: **given all the work you just planned for - is this a sustainable amount of work your team can commit to this week?**

If so, congratulations! You’ve successfully planned your week for you and your team. Everyone is in alignment on:

* How much work we’re committing to achieving by the end of the week
* What steps need to be completed (in as much detail as necessary for your team) so that anyone can grab a ticket and have enough context to complete the task at hand

If you’ve committed too much, simply drop the lowest priority work to reach your ideal work capacity. Conversely, if you don’t have enough, continue to repeat steps 4 & 5 until you reach your ideal work capacity.
]]></content:encoded>
      <dc:date>2019-11-04T13:17:19+00:00</dc:date>
    </item>
    <item>
      <title>My most referenced articles</title>
      <link>https://www.adamconrad.dev/blog/my-most-referenced-articles/</link>
      <description><![CDATA[I’ve noticed I keep referencing back to certain articles over and over again. Here’s a list of articles or videos I find myself constantly referring to even way after it was first published:

]]></description>
      <pubDate>Tue, 08 Oct 2019 22:51:57 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/my-most-referenced-articles/</guid>
      <content:encoded><![CDATA[I've noticed I keep referencing back to certain articles over and over again. Here's a list of articles or videos I find myself constantly referring to even way after it was first published:

* [Back to basics - SOLID](https://thoughtbot.com/blog/back-to-basics-solid) - This article is just so good at explaining some of the more useful design patterns you can use in web development. I probably refer to this article _at least_ twice a month.
* [Magic numbers](https://en.wikipedia.org/wiki/Magic_number_(programming)) - I feel like when I ask "is that a magic number?" people look at me like I have 3 heads. So I keep having to pull this one to remind people not to use magic numbers and yes they are a real thing.
* [Questions for our first 1:1](https://larahogan.me/blog/first-one-on-one-questions/) - For new engineering managers under my wing or whenever I run out of things to ask I keep coming back to this article because it's great advice (and it has a Google Doc template which I appreciate).
* [How to start a startup](https://www.youtube.com/channel/UCxIJaCMEptJjxmmQgGFsnCg) - I think I've watched this, no joke, 5 times for all of the lectures. I want this drilled into my brain forever.
* [The Future of Programming](https://www.youtube.com/watch?v=ecIWPzGEbFc) - Another talk I've watched so many times I hope to someday memorize it.
* [Systems Design Interview series](https://www.youtube.com/watch?v=quLrc3PbuIw&list=PLMCXHnjXnTnvo6alSjVkgxV-VH6EPyvoX) - I keep watching these because I firmly believe as an engineering manager this is one of those things that is still technical that I need to master.
* [Sizing engineering teams](https://lethain.com/sizing-engineering-teams/) - If I ever need to think about how I'm going to restructure my teams, this is the article I reference. Every. Single. Time.
* [The leadership library for engineers](https://leadership-library.dev/The-Leadership-Library-for-Engineers-c3a6bf9482a74fffa5b8c0e85ea5014a) - I'm an engineering leader. And I love to learn. Here's enough learning to melt your face.
* [Awesome Elixir GitHub project](https://github.com/h4cc/awesome-elixir) - I write a lot of Elixir for fun. All of my side projects use it. If I need a plugin or a resource, I start here first.
* [Motherfucking Website](http://motherfuckingwebsite.com/) - When people want an example of what perfect looks like, I show them this. Simple, responsive, accessible, and fast. What more do you _actually_ want? Sometimes I think I should just make my website look like this because it's what I want to do even though I know I sort of need to do my own blog design otherwise people wouldn't think I have a front end background. But I can dream I suppose...
]]></content:encoded>
      <dc:date>2019-10-08T22:51:57+00:00</dc:date>
    </item>
    <item>
      <title>Building psychological safety from scratch</title>
      <link>https://www.adamconrad.dev/blog/building-psychological-safety-from-scratch/</link>
      <description><![CDATA[Most people leave their job because of their manager. They leave their manager because they are not engaged in their work. So how do you, as a leader, engage your employees?

]]></description>
      <pubDate>Tue, 03 Sep 2019 22:51:57 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/building-psychological-safety-from-scratch/</guid>
      <content:encoded><![CDATA[Most people leave their job because of [their manager](https://www.forbes.com/sites/victorlipman/2015/08/04/people-leave-managers-not-companies/#1000172847a9). They leave their manager because they are not engaged in their work. So how do you, as a leader, engage your employees?

You might think it is giving them interesting work. You might think it is giving them lots of money and paying at the top of the industry. You might also think it is free lunch, free massages, and other cool perks you saw some Silicon Valley company offering their employees.

You'd be wrong.

**The single best way to engage your employees is to give them an environment of trust.** This means trusting them from day one, allowing them to be transparent with you, and enabling them to spread that culture to their peers.

Another way of saying this: provide psychological safety. And here's how you start doing it today.

## Inclusion

The vital first step is to make everyone a part of the conversation. It doesn't matter if you are the first engineering hire or your first day on the job. Every person needs equal _and equitable_ footing to speak and share their ideas.

Remote employees tend to get marginalized in this step. You might think you're giving everyone a voice but if you have a headquarters where everyone gathers by a water cooler to talk shop then you are leaving your remote employees out of the conversation.

Junior, less experienced employees are often thought to be included because you might go around the table to ask for people's opinions but they are still neglected because they aren't taken as seriously or with equal gravity to their more senior counterparts. And while there is something to be said for experience driving more sound decisions that shouldn't exclude a great idea simply because someone is brand new to the industry.

Give everyone a voice and make them equally as loud.

## Safety to learn and contribute

Google made a huge stir about 20% time; that you could take a whole day out of your week to do whatever you want to develop your career outside of your regular value-added work to the company.

Since when is software engineering a career where you only need to learn 20% of the time? **Learning should be an integral, continuous part of the software engineer's professional experience**. And the biggest part of learning is trying new things, not getting them right the first time, and persevering to get back up and try again.

I just watched a great [TED talk](https://www.youtube.com/watch?v=9vJRopau0g0) on how just giving children encouragement towards their goals gave them a _16% increase in their chance of success_. How much would you pay for 16% fewer bugs, 16% more output, and 16% more engagement from your employees? Well mail me your checks because you can do it all for free if you simply encourage a culture that **failure is a gift** and not something to be reprimanded for.

We learn when we fail. We also learn when we share that failure (and subsequent success) with our teammates and greater community. Not only do we need a platform to try and fail, we also need a platform to share our findings along the way. Being able to contribute to the greater community is another vital point of psychological safety.

In my teams, **we acknowledge that even updates to documentation are contributions**. I've met plenty of high achievers who are annoyed that they can't contribute to code on day one. I tell them that if the README is checked into the code base, and you find a flaw in our setup README docs, then that is checking in code. **All contributions are welcome.**

When you know that your README change was as valuable to the team as some hardcore algorithmic update to some C++ code, you remove the barrier to entry that only the most senior folks can make the most outstanding contributions. **Outstanding contributions come from outstanding people.**

## Safety to challenge the status quo

No one wants to rock the boat on their first day at the job. That's why new hires are the only ones wearing button down shirts with collars. Everyone _thinks_ they have to dress to impress and start off on the right foot and by the end of the week these same engineers are wearing t-shirts like the rest of us.

What if, like me, they just liked getting dressed up because they enjoy dressing up? Or what if you didn't think that having to be in the office was required for productivity in the first place?

Having radical ideas is the basis for innovation. If you couldn't challenge societal norms then the founders of Airbnb would have never been able to convince people that it's totally okay to sleep on a strangers couch and that everyone would be doing it a decade later. If you asked anyone in 2004 that premise would not only sound absurd but downright dangerous.

Why are only the successful entrepreneurs allowed to innovate? Why can't it be every single person in your group? The only barrier to this hurdle is yourself. There is literally _nothing_ stopping you from having anyone challenging the status quo. All you have to do is _let go_. Control is hard for some folks to relinquish but it is essential if you want to get this part of psychological safety.

## No fear

Which leads me to my last point: no fear. Letting go of control is based on a fear you have that if you don't have control then bad things will happen. A fear-based mentality will only get you so far. It is self-limiting in nature.

A fear-based culture leads to people feeling:

* embarrassed
* marginalized
* threatened
* punished

A fearless culture, meanwhile, leads to people feeling:

* celebrated
* included
* empowered
* rewarded

Which feelings do you want to feel?

I hope I've convinced you that psychological safety is the bedrock for high-performing teams. A fearless culture leads to greater productivity, higher engagement, and increased satisfaction at work. And it costs you nothing but relinquishing a bit of your control freak instincts to trust your group to do the right thing.

If you can take that first fearless leap towards a safe environment then you will be rewarded with all of the benefits that come along with it.
]]></content:encoded>
      <dc:date>2019-09-03T22:51:57+00:00</dc:date>
    </item>
    <item>
      <title>Anticipating needs is a great way to manage up</title>
      <link>https://www.adamconrad.dev/blog/anticipating-needs-is-a-great-way-to-manage-up/</link>
      <description><![CDATA[You need to anticipate your manager’s needs. This is crucial for success in any job because making your boss’s life easier is one of the best ways of getting their attention about the value you bring to the company.

]]></description>
      <pubDate>Mon, 19 Aug 2019 12:19:06 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/anticipating-needs-is-a-great-way-to-manage-up/</guid>
      <content:encoded><![CDATA[You need to anticipate your manager's needs. This is crucial for success in any job because making your boss's life easier is one of the best ways of getting their attention about the value you bring to the company.

## Visibility

The first thing you might be thinking is "what's in it for me to try and do my manager's job?" Start by flipping the script: if you have direct reports (or imagine that you do), how would you feel if one of them approached you to lighten the load or make you look better? It's happened to me before and I can tell you it's both relieving and flattering.

When you take [extreme ownership](https://www.amazon.com/Extreme-Ownership-U-S-Navy-SEALs/dp/1250067057) over your career it can be surprising and liberating to have someone else want to own things for you. It's also proof that your system of delegation and ownership is working when others look to your example and want to emulate and uplift the whole organization. So if we go back to your situation and you help out your boss you should expect welcome relief and joy that you are checking in on them.

## Extreme ownership

Now you understand the value of helping out your manager but you still haven't mastered extreme ownership. What is it and why can it help you?

As I linked above, you can read the book for a series of anecdotes or I can summarize it as this:

> Extreme ownership is owning up to everything within your control; the good _and_ the bad.

Your boss is concerned your team is going to miss the deadline. You believe it is because there is a developer who isn't pulling their weight.

Wrong.

When you do a root cause analysis of why the developer isn't pulling their weight, you can always trace it back to _you_. You didn't set expectations for delivery on the deadline. You didn't check in every week to see if all team members were up to speed and unblocked. You didn't provide training or education for those falling behind who needed more assistance. If something goes wrong within your org then there is always more you can do to improve the outcome.

I have heard this advice paraphrased many times since becoming a manager and it is that your team gets credit for all of their success and you credit credit for all of their failures. When your team launches a big app you congratulate your team's tireless efforts in front of the whole company. When a security vulnerability loses you a customer then you take one on the chin in front of your senior vice president. This is the price you pay for moving up the management ladder.

## The key to moving up

Beyond staying humble the real reason is around your position. If you run a team by definition you're already celebrated for being a leader with your title and salary. What more do you need? **You need stellar employees who like working for you.** In order to maintain that title, salary, and status, you need to be seen as a leader worthy of more directs and more responsibility. The only way to do that is to enable successful teams who like working for you.

No one wants to promote a leader to a senior leader who can't retain their employees or get the job done. And the only way to do that is to make your teams awesome. One _really_ easy way to help do that is to take flak when things go wrong and prop them up when they succeed.

## An example: organizational design

One night I was curious about how my teams would [scale](https://lethain.com/sizing-engineering-teams/) over the next year or two so I played around with Excel for maybe 20-30 minutes. It wasn't a lot of time and I had a mental itch I needed to scratch.

The next day in my 1-1 with my manager I decided to show him both my short-term team plan over the next 3 months as well as the 1+ year plan. We had nothing on our agenda about this and we won't even begin to do budgetary talks until the end of the year.

Even with all of those considerations my manager was happy to see it. The real eye opener was that a week later in standup with his direct reports (all directors), he announced all teams would begin to work with him on how their teams would evolve over the next several months/years.

So you tell me: who has the leg up in the conversation? Who spurned this idea? If you're thinking it's me because of my proactive work with the 1-1 I'd say you're right. While I can't confirm for sure, it is awfully circumstantial that a week later my topic of long-term organizational design was to be a central topic for all of my peers to my manager over the next several weeks.

That was the singular moment when I realized the power of anticipating needs and being ahead of the pack is a powerful force in your career.

## Armchair quarterbacking and moving forward

Would I do anything differently? **I would have started sooner** so hopefully I can make that happen for you. Moving forward, this will continuously be a part of my duties particularly when I have more room on my plate for additional capacity. Even if it doesn't revolve around your manager's needs you should try and help others because it's the right thing to do. If you're floating around the pool and someone else is drowning it's only instinct to jump in and help them out; this should be no different while you're working. We should all be doing more to help others when we are in a position to assist so go ahead and do it!
]]></content:encoded>
      <dc:date>2019-08-19T12:19:06+00:00</dc:date>
    </item>
    <item>
      <title>A 90 day plan for managers</title>
      <link>https://www.adamconrad.dev/blog/a-90-day-plan-for-managers/</link>
      <description><![CDATA[Whenever I’ve been promoted or moved into a management position, I reach for this book which I learned about after reading this post on how your first three months are so important for setting the right tone with your manager and the role you are signing up for.

]]></description>
      <pubDate>Wed, 24 Jul 2019 12:19:06 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/a-90-day-plan-for-managers/</guid>
      <content:encoded><![CDATA[Whenever I've been promoted or moved into a management position, I reach for [this book](https://www.amazon.com/First-90-Days-Strategies-Expanded/dp/1422188612) which I learned about after reading [this post](https://hbr.org/2009/03/why-the-first-100-days-matters) on how your first three months are so important for setting the right tone with your manager and the role you are signing up for.

I'm going to walk you through the 90 day plan I used for my current jump into director that is based off of this research so you can implement something like that for yourself if you are lucky to be gifted a newly-minted management position.

## Days 1-30: Evaluation

The first 90 days are important. By extension, the first 30 of those 90 days are the most important.

The reason is because first impressions are everything. When you land a new role you may not be new to your company (or maybe you are) but you are new to your team _in this role_. It is therefore vital to your success to spend these days getting to know your team(s).

**Start by setting up 1-1s up and down.** These should be with every direct, every week, for at least 30 minutes per session. No exceptions. You'll also want to make sure your 1-1 with your manager (and a skip-level with your manager's manager) is set up as well. These are necessary not only for building relationships and rapport but also for information gathering and learning the lay of the land.

**Set and get expectations.** Your first 1-1 with your manager should tell you why you're here and what is expected of you. Once you know that, you should be able to pass down and delegate those expectations to your directs. Knowing where you stand and what you need to do ensures what you believe your role is about matches what your manager (and your company) believes what your role is about.

**Assess your product.** While you assess your people you also need to assess their output: the product you are developing. Given my audience I'm going to assume you're developing some sort of digital distributed product like a SaaS web app. This means quite a few things:

1. Can you pull down, build, and run your code? Can you run the tests that come with it?
2. Once it is running, can you log in, sign up, and do the things your customers do?
3. How long are things taking? Can you navigate your app quickly?
4. Do you run into bugs? How frequently? Do you have a lot of test coverage to catch errors?
5. If you change languages, time zones, or use a screen reader, can you still use your application? What if you turn off JavaScript?
6. Are there any ways to perform unsafe activities? What if you pasted code into a feedback form or tried to make unauthorized server calls from a curl command?

These are just a few ideas I came up with off of the top of my head. The important thing is you do the digging yourself and assess what you think of the thing you are responsible for. If there's a problem, is it with the team or the product itself? Is it fixable? We'll dive into this piece after the first month.

## Days 31-60: Recommendation

Now that you know what you are dealing with your next step is to make recommendations to your manager as to what to do next. If you're in a highly autonomous organization you may only need to recommend for the sake of posterity but it is still a useful exercise to make sure your consultation is thorough.

I split my recommendations into three major buckets.

### Team recommendations

This bucket is about stack ranking your team. Some people might be put off by this but the truth is you will naturally discover who are your star players and who you could do without. Whether you want to bucket them consciously or subconsciously you're naturally going to do this anyway so it is better to accept that and get ahead of it.

Once you know that you can look at their roadmaps and how well their are at estimating their efforts. For that we move into process.

### Process recommendations

What is their process for reaching those estimates? Do they use story estimates based on the Fibonacci sequence? Do they t-shirt size bigger chunks of work? Are those terms completely foreign to your teams and just wing it? You want to know what their process is (or lack thereof) and if there are any inefficiencies with their approach.

Given you are a new manager for this group there almost certainly are some process deficiencies otherwise they wouldn't hire you to take over this team. **This may be one of the most key levers you can pull to make a great first impression.** You may not be able to hire or fire anyone right away but you can change how this current team works.

Some ideas I would introduce at this point are:

* Feedback in your 1-1s
* Adopting some kind of software development lifecycle (such as the Google Design Sprint)
* Standardizing individual and team meetings
* Setting your teams' mission, purpose, values, and goals
* Building transparency through heavy use of documentation and over-communication

Many companies lack these basics so any opportunity you tackle low hanging fruit will be easy for you to implement and also make you look really good and productive in your first few weeks. From here you might even be able to extract some practices that might be useful to groups outside of your organization which is where the last part comes in.

### Engineering recommendations

I would hazard a guess that one or more of your recommendations are not only new to your teams but new to your organization if not the entire company. These are your highest-lever recommendations that will have the biggest impact on the business. **Promote the heck out of these.**

Start by creating a proof of concept within your teams. Measure them over the next several weeks. Any sort of sizeable improvement should be lauded and promoted to other groups to try out. Continue to measure and repeat to see if your success is magnified. If so, you're onto something really great.

Tools and coding standards are great opportunities I've found in my first 60 days. I noticed our code had one tool to handle code coverage and another tool to handle code quality metrics like [cyclomatic complexity](https://en.wikipedia.org/wiki/Cyclomatic_complexity). I showed our company a tool that did both and cost less. How could you say no to that? Which leads me to the last 30 of these 90 days: actually getting things done.

## Days 61-90: Implementation

You've assessed your group. You've recommended how you need to steer the ship. Now you've actually got to grab the wheel and take charge! The first 90 days only matters if you turn ideas into reality.

I've seen people make the mistake of dragging out the assessment and recommendations (or even just the assessment) across the whole 90 days. **Without anything to actually show for your business in terms of value-add to customers is like hitting a triple in baseball only to have the next three batters strike out.** A triple is great, don't get me wrong, but you can hit all the triples you want and still never win a single game if you don't get to home plate and score some points.

In other words, **you need to take all of the word and advice you've collected and show some results**. People expect a ramp up period so no one is expecting high-output from day one. But if you really want to cement you first 90 days as something to celebrate then you need to put the pedal to the metal. As a director, I see my primary tasks manifest in 6 ways:

1. Hiring
2. Career development
3. Project management
4. Running retrospectives
5. Analyzing engineering metrics
6. Architecture/Systems design

Your recommendations should influence one of your primary tasks. If they don't then you aren't acting on them and they aren't worth executing. There is no better way to cap off the first three months of your new role with a series of wins under your belt.

Discover what your primary tasks are (which you should have discovered when you first had expectations set with your manager) and **implement your recommendations into duties of your job daily work**. Once you have your wins it get the ball rolling for the tasks you need to perform in the next 90 days and the 90 days after that. Soon enough you'll be gunning for the next promotion and you'll start the process all over again and I look forward to the next time you read this article.
]]></content:encoded>
      <dc:date>2019-07-24T12:19:06+00:00</dc:date>
    </item>
    <item>
      <title>How to justify your cost to a client</title>
      <link>https://www.adamconrad.dev/blog/how-to-justify-your-cost-to-a-client/</link>
      <description><![CDATA[
  TLDR: Price yourself in a way that feels low to the client even if it is high in absolute terms.


]]></description>
      <pubDate>Thu, 06 Jun 2019 21:50:40 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/how-to-justify-your-cost-to-a-client/</guid>
      <content:encoded><![CDATA[> **TLDR: Price yourself in a way that _feels_ low to the client even if it is high in absolute terms.**

If you're new to freelancing or [consulting](https://anonconsulting.com/) the advice that gets passed around a lot is to not charge by the hour because then you're making yourself a commodity. That instead you should use [value-based pricing](https://training.kalzumeus.com/newsletters/archive/consulting_1). In software, this is flawed for 2 reasons:

1. **Most software gigs are longer-term.** Marketers and designers can easily charge value-based pricing because their projects or campaigns are extremely finite. A campaign or a logo are just that - you might iterate a few times to get what you want but once you get the deliverable that's it. Even if you do a shorter 4-6 week engagement as a software developer you will inevitably have a bug or security update that will require maintenance after the engagement is over; it just doesn't "end" when you deliver the code.
2. **Most companies don't price software products; they price developers.** Companies don't hire freelance designers, they buy logos. Companies don't hire marketers, they buy campaigns. But _everyone_ hires software developers. They don't want to buy a SaaS app from you. They want to buy your time. So until the industry moves away from buying your time, the vast majority of clients will still value you as a dollar-per-hour commodity.

I know this having tried to win plenty of clients over with this approach. I also know this now having working at a company that hires lots of contractors.

So let's dive into the 5 Whys to see if we can find a root cause and a solution.

* **Why doesn't value-based consulting work?** Because companies still buy a software developer's _time_ and not what they _produce_.
* **Why can't we sell software and not ourselves?** Because software is a blurred line. Unless you are expected to write the entire application for a client what is your code versus someone else's is very hard for the client to define.
* **Why can't we clearly define the products we build?** Because software is built with teams. If you are trying to find clients chances are you are not their first (or their last) developer. Software is very difficult to get done as a solo project and is rarely done in a vacuum. Software is a team sport.
* **Why is software a team sport?** Because it's super hard!
* **Why is software hard?** Because we are trying to map the decisions of the human mind onto a flat glowing metal screen. How easy could it possibly be to describe the 300 billion neurons of the human brain?

So software is really hard to do on your own and therefore hard to distinguish your software product from the rest of the team you are joining as a contractor. Does that mean you should throw in the towel and give up? **No way!**

Yes, freelancing and consulting is super hard and unless you have the stomach to sell yourself and get lots of rejections you are going to have a bad time.

But there is one thing you can do to overcome the hurdle of a client justifying the cost of going with a contractor: **convert your desired value-based outcome into an hourly rate.**

## Estimates are your saving grace

The key to make this all work is to work backwards from your ideal to a realistic estimate. This is best illustrated with an example.

Suppose you have a project that you think is worth $100,000. You want to tell the client the charge is $100,000 because of blah blah something valuable. Now ask yourself how long do you think this will take? 10 weeks? That's about 400 hours or $250/hour which is pretty dang good.

The problem is that no one is going to pay $250/hour for a contract developer. Unless you work for a prestigious, well-established consulting firm, you are going to have a hard (if not impossible) time finding clients that will hire independent consultants at that rate.

**Here's the key:** if you tell your client you charge $100/hr now they're listening. It just means to get to that rate you would theoretically need to do 1,000 hours worth of work. **But your client doesn't care if it takes you 1,000 hours or 400 hours if they agree to pay you to complete the project in the time you've told them it would take.**

So _if you end up only taking 400 hours to complete the project_ you've upped your hourly rate to the desired rate for your original project _and_ the customer gets their project 60% faster than they thought they would have. You think they are going to complain if their critical project arrives early?

Isn't that [inflating your hours](https://en.wikipedia.org/wiki/Parkinson%27s_law)? On the contrary, there is an [entire law](https://en.wikipedia.org/wiki/Hofstadter%27s_law) devoted to a programmer's hubris in estimating software.

Your ideal hourly rate is probably too optimistic. A heavily-inflated hourly estimate is based on something like Hofstadter's Law to compensate for the things you don't know.

In the end, your actual hours billed in this example will probably fall somewhere in the middle around 700-750 hours because of things you failed to account for. Either way, you're still making more than the hourly rate you billed and your client still gets their project delivered on time or earlier.

Next time your client balks at your price remember Hofstadter's Law and buffer in quite a bit of extra time so you can land the project early, make more per hour, and use the rest of that time to find more clients and bill more hours.
]]></content:encoded>
      <dc:date>2019-06-06T21:50:40+00:00</dc:date>
    </item>
    <item>
      <title>A plan for finding clients quickly</title>
      <link>https://www.adamconrad.dev/blog/a-plan-for-finding-clients-quickly/</link>
      <description><![CDATA[This time last year I was scrambling.

]]></description>
      <pubDate>Thu, 06 Jun 2019 21:50:40 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/a-plan-for-finding-clients-quickly/</guid>
      <content:encoded><![CDATA[This time last year I was scrambling.

A month before everything was going great. I had an awesome client, charging (what I thought was) a great rate; no complaints.

And then I got the email.

The client was terminating the project early and I had to wrap up in the next 30 days. It had nothing to do with my personal performance it's just the company decided they couldn't afford to spend this much on software so they cut the funding abruptly.

What was I to do?

Extreme circumstances call for extreme measures. I felt a lot of personal pressure to not have a gap in clients. The result: I landed a new client without any gap in employment. And I can safely say I've never had a gap with clients in my entire time as a full-time freelance consultant developer. How did I do it?

One word: process.

## The client prospecting process

When your back is up against the wall you will do what it takes to make things work. I will admit this is a very aggressive course for grabbing clients but it works if you're willing to put in the work. For every week day in the 30 day window I had, this is what I did:

### Reach out to 5 contacts in my network

The first step is warm outreach. Cold outreach is a dead end in such a short time frame to it is vital to reach out to folks in your network. So I start by asking them if their company needs any contracting work. If they don't, can they recommend 3 people who might even remotely need it _or_ if they ever worked with consultants before and used a firm.

### If there are leads, follow up

After a few days I was able to build up a consistent rotation of 3 leads to prospect. This is where you get to a conversation in person or over the phone as soon as possible. **Any hesitation leads to indecision and ultimately no purchase of your services.** Time decay is a real thing so you must apply that pressure to get on the phone.

### If there are prospects, promise them a 24 hr turnaround

I have to admit that some of my favorite parts of the prospecting process are right after that intro phone call. This is a part of the process that can get you into trouble because you're carrying a lot of momentum and a lot of positive energy without any commitment. And those good vibes can cloud your judgement.

This is why it is crucial to stay focused on getting to closing by not celebrating a good conversation and just **getting things done with a fast estimate**. Your prospecting call should leave you with a bunch of notes to address so get them a statement-of-work/proposal in 24 hours.

Again, time decay is real so every hour you waste is time for the prospect to decide they don't need to go with you. Building trust is about doing what you'll say. You'd be surprised at how many people will not follow through on super simple things like this.

### If there are potential buyers, check in

At this point you're getting real close to a sale. You can taste that sweet signature but you can't slow down. You should have put in your proposal that you'll check in within 48 hours. So make sure **when you deliver your proposal you mark on calendar for 48 hours later to check in with that prospect**.

Do revisions, get on the phone; whatever it takes. The objective is to ensure that **your next client feels special and valued and you want the transaction to be mutually beneficial**.

Knowing you have your clients' best interest at heart is not only important but it is something very few consultants seem to care about anymore. Of course you want to make money and your client wants to generate revenue/save money from the software you're about to give them. But just stating that you want to go the extra mile for them is not a phrase most clients hear or that most consultants say. **You should relish in any opportunity to stand out**.

### Grab that signature - you did it!

At this point you've done everything you can. Like any funnel, most potential clients won't even respond to your initial email or LinkedIn message. Few will respond. Fewer will get on the phone. Almost none will read a contract or proposal. **But it only takes one signature to be in business.**

It kind of reminds me of all of those cheeky jokes like "what do you call someone who graduates at the bottom of their med school class? A doctor." It's true.

You can strike out thousands of times. You could be trailing the entire game. But if the clock reaches zero and you hit a homer to win the game, you're a winner.

I was able to close multiple six figure deals with this strategy and while I heard a lot of rejection at every stage of the process, the fact remains that I never had a gap in clients because I aggressively pursued this strategy. I just hope you don't have to wait until the 11th hour when your contract expires early and you're scrambling to add new clients to your pipeline. Get started on this stuff today so you're not in trouble tomorrow and happy prospecting!
]]></content:encoded>
      <dc:date>2019-06-06T21:50:40+00:00</dc:date>
    </item>
    <item>
      <title>Build your network now</title>
      <link>https://www.adamconrad.dev/blog/build-your-network-now/</link>
      <description><![CDATA[My contract got cut short. I’ve got to assemble another client in the next 30 days or the money stops coming in. Thankfully, I’ve built a strong network over the last decade I’ve been in this industry so I think I should be alright.

]]></description>
      <pubDate>Tue, 02 Apr 2019 22:49:10 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/build-your-network-now/</guid>
      <content:encoded><![CDATA[My contract got cut short. I've got to assemble another client in the next 30 days or the money stops coming in. Thankfully, I've built a strong network over the last decade I've been in this industry so I think I should be alright.

Would you be okay if this happened to you?

I've heard this phrase so many times it has been drilled into my brain when it comes to networking:

> Dig the well before you get thirsty

Most people think of networking as a _reactionary_ activity. You network with people at meetups because you need something from someone. No wonder people hate to go to meetups!

**Networking should be a _proactive_ activity.** You _offer_ help and connections. You build people up and bring them together _just because you can_.

What does this get you?

If you've asked that question you've already lost. Stop only thinking about yourself. **No one wants to network with selfish people.** The question you should be asking is:

> How can I make the world a better place for my fellow humans?

When you give back, you get more in return. And I don't mean this in the touchy-feely validating kind of way. I mean real benefits when your contract runs out and you need more money NOW kind of benefit.

Imagine your friend is looking to start a company. Imagine your other friend is looking to start a company. You see two people and one opportunity to actualize. In that moment it probably has no material impact on your life. But in the future when you are looking for a job and those two are now hiring, who do you think they are going to ask to fill their top spot? Who do you think will have their own network of folks looking for freelancers to help build the MVPs for their startups?

If you're thinking this imaginative scenario actually happened to me, you'd be right.

I actually helped connect my friends together to start a successful startup. It did nothing for my wallet for years. But eventually I ended up working for their company. And years after that they helped me land some key clients when I went into business for myself. And we're all still friends!

This stuff works. But it only works if you live with an abundance mentality that **it's not all about you**. If you only network when you're desperate and need something, you're going to have a bad time. Which is no wonder why everyone fails at it and thinks it's scummy and no fun.

If, instead, you **make networking about genuine, authentic connection**, you're apt to get everything you could ever want out of networking and way more than you bargained for. I may be a sample size of one but my hope is that you'll add to that list of anecdotes as well.
]]></content:encoded>
      <dc:date>2019-04-02T22:49:10+00:00</dc:date>
    </item>
    <item>
      <title>When to use Scrum or Kanban</title>
      <link>https://www.adamconrad.dev/blog/when-to-use-scrum-or-kanban/</link>
      <description><![CDATA[I used to believe Kanban was the solution to all project management problems. Kanban, if you don’t know, is a Japanese word that means “just-in-time” that was developed by Toyota for lean manufacturing. For software development it just means that you continuously groom tickets for priority and don’t create rigid structures around sprints (like Scrum).

]]></description>
      <pubDate>Wed, 06 Mar 2019 12:05:49 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/when-to-use-scrum-or-kanban/</guid>
      <content:encoded><![CDATA[I used to believe Kanban was the solution to all project management problems. Kanban, if you don't know, is a Japanese word that means "just-in-time" that was developed by Toyota for lean manufacturing. For software development it just means that you continuously groom tickets for priority and don't create rigid structures around sprints (like Scrum).

This made a lot of sense to me because I work in web development where the web has no gates on pushing changes and companies continue to innovate faster and faster. How could any other methodology be any better? It turns out I was wrong and you may be wrong too if you think that Scrum has to be inferior in all scenarios. I now use both methodologies for different purposes.

## When to use Kanban

Kanban is still great. The central tenants of Kanban are:

* Visualizing your work with a progress board
* Limiting work in progress to reduce time to completion
* Maximizing developer flow
* Continuous delivery and therefore continuous change

These objectives work for certain products and teams. I will use Kanban when:

* **We're building a web product.** Nothing is federated and you can push to the web at any time.
* **The first version of the product is already live.** It's easier to think in terms of continuous change when you are going from 1 to _n_ instead of 0 to 1.
* **We don't have rigid deadlines.** If everything is continuous then each push to production is in essence a new version of the product for your customers to gain value from. If you don't need to "release" a collection of features then each push adds value.

If you work at a modern web company with a SaaS product you'd probably stand to benefit from the Kanban framework. Living in this bubble I forgot that many software companies don't fit this mould. Lots of teams operate in a different way.

## When to use Scrum

In comparison, Scrum's primary objectives include:

* Commitment to deliver your work through sprint intervals
* Continuous learning and refinement with each sprint
* Ceremony and roles to increase teamwork
* Velocity defines refinement strategies
* Change comes between sprints

Here are a few scenarios where Scrum will be superior to Kanban:

* **We're building a mobile product.** Mobile is federated by app store releases. As wild as this sounds, it can actually be detrimental to release mobile software too frequently. The app stores think your software is to buggy and your customers hate to keep downloading your new releases.
* **We need to build a concrete MVP.** When you have a clear 0 to 1 moment (i.e. going from nothing to something) it's helpful to have a north star separated by a series of sprints. There is no continuous delivery element here because no delivery prior to the MVP adds value to your customers.
* **We have a deadline we have to work back from.** This is particularly relevant for developers in the gaming industry. You need to increase velocity and you have to reach a deadline with enough buffer to make sure you don't miss.

At Indigo we use Scrum for our mobile teams because we know we don't want to push to the app store more frequently than once every two weeks. This aligns nicely with the common two week sprint cadence for the Scrum framework.

## When to blend both

You can take elements of both and create a blend of the two frameworks. You can organize projects into sprints but make sure you limit work in progress. You can also use Kanban boards with set releases with changes mid sprint. Ultimately, you can do whatever you want as long as **it works for your team**. I don't know of other modern frameworks for project management so if I'm missing one that has you captivated please [let me know](https://twitter.com/theadamconrad).
]]></content:encoded>
      <dc:date>2019-03-06T12:05:49+00:00</dc:date>
    </item>
    <item>
      <title>How to demonstrate impact</title>
      <link>https://www.adamconrad.dev/blog/how-to-demonstrate-impact/</link>
      <description><![CDATA[There are two fundamental ways to demonstrate impact for your company: generate revenue or cut costs. How do you measure that impact as a software engineer?

]]></description>
      <pubDate>Sun, 10 Feb 2019 23:54:49 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/how-to-demonstrate-impact/</guid>
      <content:encoded><![CDATA[There are two fundamental ways to demonstrate impact for your company: **generate revenue or cut costs**. How do you measure that impact as a software engineer?

## Ways you can generate revenue

The advice I've always heard is to be part of a revenue-generating division in a company. While engineering on the whole is a cost center, the easy way to tell if you are on the revenue side is asking yourself **"does my product make money?"** If the answer is an emphatic _yes_ then you're good. Usually product teams are revenue generators. As an individual, here are the ways you add revenue to your bottom line:

* Measure the value of the work you do. Does your new product offering increase conversions? By how much? How much is a conversion worth?
* Up-sell, cross-sell, and ensure that all of your products are easy to find
* Make your application [fast](https://www.gigaspaces.com/blog/amazon-found-every-100ms-of-latency-cost-them-1-in-sales/) and [accessible](https://www.cdc.gov/ncbddd/disabilityandhealth/infographic-disability-impacts-all.html)
* Take a tour of customer service by answering emails/calls - not only will you improve customer relations you will likely source new ideas for products or improving existing ones
* Participate in hackathons - famous examples of product innovations include the [Facebook like button](https://en.wikipedia.org/wiki/Hackathon), PhoneGap and the entire company of GroupMe (which was later bought by Skype and then Microsoft)

## Ways you can cut costs

If you aren't adding revenue then the next best thing is to cut costs. If you work on R&D, infrastructure, DevOps, or IT, then you are part of a cost center and your focus should be on cutting costs. Here are ways you can cut costs as an individual engineer:

* Find a cheaper alternative for a tool you already use
* Build an in-house replacement for something you use as a tool
* Take over something that is being maintained by consultants (i.e. consultants are _expensive_)
* Say "no" to meetings without agendas, goals, and action items
* Use automation tools that save you significant amounts of time
]]></content:encoded>
      <dc:date>2019-02-10T23:54:49+00:00</dc:date>
    </item>
    <item>
      <title>Teamwork makes the dream work</title>
      <link>https://www.adamconrad.dev/blog/teamwork-makes-the-dream-work/</link>
      <description><![CDATA[This is a phrase my best friend tells me all of the time.

]]></description>
      <pubDate>Wed, 09 Jan 2019 22:44:58 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/teamwork-makes-the-dream-work/</guid>
      <content:encoded><![CDATA[This is a phrase my best friend tells me all of the time.

The first time I heard it I thought it was cute but over time it has become downright pithy and succinctly describes how I think about building teams.

For my teams, one of our top values is teamwork. It means putting the team before yourself. It also means giving yourself permission to rely on your team to get things done. Heroes are _not_ welcome.

Heroes, rock star programmers, ninjas...whatever you call them, these are the kinds of people who pull all-nighters and get lots of things done. They "save" projects. But at what cost?

The cost is alienation. They tend to think that since they saved the project by hitting the deadline that they are owed something by the rest of the team. They also tend to bring a "holier than thou" attitude going forward.

And while people may applaud the rock star for their performance it does not encourage them to work well with the rest of the team; if anything, it encourages rock stars to continue to fly solo because the rest of the team just gets in their way.

Have you ever wondered why sports teams only have one or two all-stars? Why doesn't LeBron James team up with Kevin Durant and Carmelo Anthony? Why not have Tom Brady gang up with OBJ and every awesome player?

They actually do this once a year and it's called the All-Star Game. Every sport has it and guess what: virtually no one watches it. Why is that? Beyond the fact that it's an exhibition game with no real consequence, I think another underlying issue is that **a bunch of divas don't play nice with each other**. If you have a whole team of all-stars then no one is a stand out and they _hate_ not being the stand out. The irony of having great players on a team is that if they can't actually be great they don't really want to play for a team.

**The best players are still at the mercy of the teams they play for or else they don't have a platform to stand out.** This is a powerful realization if you have a rock star programmer on your team. As much as they may whine and moan at the incompetence of their teams from time to time, it is the existence of that team that allows them to stand out. If they don't wise up to meeting the needs of the team then _they_ fail to actualize their own true potential.

This is why the **team always wins over the individual**. Because even if your interests are purely selfish and all you want to do is write great code with no one in your way, the only way you can make that happen in a successful company is to **build up and support a successful organization around you**.
]]></content:encoded>
      <dc:date>2019-01-09T22:44:58+00:00</dc:date>
    </item>
    <item>
      <title>Ending Our Algorithmic Journey by Proving NP-Hardness</title>
      <link>https://www.adamconrad.dev/blog/np-completeness/</link>
      <description><![CDATA[Proving the hardness of an algorithm for non-polynomial algorithms takes skill. As we wrap up our journey of The Algorithm Design Manual we’re going to leave you with some final tips to help prove the hardness of these intractable problems. But first, onto the Daily Problem:

]]></description>
      <pubDate>Thu, 22 Nov 2018 20:17:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/np-completeness/</guid>
      <content:encoded><![CDATA[Proving the hardness of an algorithm for non-polynomial algorithms takes skill. As we wrap up our journey of _The Algorithm Design Manual_ we're going to leave you with some final tips to help prove the hardness of these intractable problems. But first, onto the Daily Problem:

> This article is part of the [Data Structures and Algorithms Series](/tag/algorithms/) following the course [The Analysis of Algorithms](https://www3.cs.stonybrook.edu/~skiena/373/). If you missed the [previous article](/blog/hard-reductions/), check that out first as well as [a copy of _The Algorithm Design Manual_](https://www.amazon.com/Algorithm-Design-Manual-Steven-Skiena/dp/1849967202).

## Answers to the previous Daily Problem

> Show that the following problem is NP-complete: for a graph `G` with integers `k` and `y`, does `G` contain a dense subgraph with exactly `k` vertices and at least `y` edges?

When I look at this problem I instantly think of the tree of reductions that we went over the last few articles. So right away I'm thinking of the graph-specific configurations: independent set, vertex cover, and clique. The key in choosing these three is in the word _dense_...which of the graph types in the reductions list deals with dense configurations? My first instinct is clique because of how we want every vertex to touch every other node (think back to the example of the pentagram in the pentagon).

So if we can reduce a dense subgraph into a clique via a reduction, and we know that a clique is part of the NP-complete hardness tree, then we have our solution.

A clique is a fully-connected graph. So if the dense subgraph is a subset of the connections in a clique, your dense subgraph generalizes to clique and therefore the graph is also NP-complete. This is called a [K-clique densest subgraph](https://en.wikipedia.org/wiki/Dense_subgraph) if you want to know more about the formal proof.

## Nearing the end: tips for proving hardness

We've come a _long_ way in our journey in understanding data structures and algorithms. At this point we're entering territory where programming languages no longer matter; these theoretical algorithms have no real solutions so we can't map them to code samples. Whether you're studying this for fun or for interviews you may reach this final stage where you need to understand how to solve for uncharted territory. Here's a few tips on proving algorithmic hardness:

### Choose a good, simple NP-hard graph

The first and most important thing is to identify the right kind of NP-complete problem. Knowing the chart of problem translations is extremely handy in these situations. Here's a few based on how much you think you can fit in your head:

* [A simple one](https://upload.wikimedia.org/wikipedia/commons/8/89/Relative_NPC_chart.svg)
* [A bigger graph](https://www.edwardtufte.com/bboard/images/0003Nw-8838.png)
* [The motherload of current hard problems](https://adriann.github.io/npc/dot.png)

If you look at the Daily Problem, much of the effort came down to choosing the right source problem. Clique was clearly the right choice for K-clique densest subgraph. But even then, you could still find a reduction using vertex cover or 3-SAT, it would just be harder to make the translation from the source to the destination problem. For most of these problems, you pretty much only ever need to **start with 3-SAT, vertex cover, Hamiltonian path, or integer partition**:

* Worried about order or scheduling like a calendar? Start with _Hamiltonian path_.
* Dealing with selection and filtering? Start with _vertex cover_, though _clique_ and _independent set_ are obvious offshoots if you can see the distinction.
* How about combinatorics or large numbers? _Integer partition_ is what you want.
* For everything else? _3-SAT_ is at the top of the NP-complete food chain and is extremely versatile.

### Constrain the problem as much as possible

Constraints are good. An undirected graph becomes a directed graph. An unweighted graph adds weights. Or we can flip it: we did this when reducing TSP into a Hamiltonian cycle by turning weighted edges into unweighted ones, equalizing the cost of each edge for a Hamiltonian cycle.

Your constraints should inject harsh penalties for deviating away from your reduction. When we reduced Integer programming into 3-SAT, the maximization function and target value all became inconsequential. To ensure we didn't let that get in our way, we had to neutralize the maximization function to the identity function and set the target to `0`. Any other set of values would make it more difficult to prove.

Once you have the constraints, _force a decision_. Make it so that the choice is between A or B and never A _and_ B.

### Reductions are easier than algorithms but keep your options open

When your head is in the NP-complete or NP-hard space you automatically assume that the problem is unsolveable with an algorithm. However, non-polynomial problems in the exponential space can still be solved with techniques like dynamic programming to bring it back down into efficient polynomial space.

In other cases, you can use approximation algorithms with heuristics to turn NP-hard problems into polynomial-time solutions. Maximum clique, minimum vertex cover, and the Knapsack problem all benefit from approximation algorithms to arrive at a relatively correct but efficient result.

## Final thoughts

This project of auditing the class for _The Algorithm Design Manual_ was truly a labor of love. Learning by teaching is a great way to really cement your understanding of the materials in this series.

If you used this to help you understand data structures and algorithms for an upcoming interview, I want to leave you with a selection of problems over the whole book as sort of a final exam to test your knowledge. These problems are specifically designed for those interviewing at companies, so if you can tackle these, you'll be in a great spot. Good luck and thanks for sticking with me!

1. Problem 1-29.
2. Problem 1-31.
3. Problem 1-33.
4. Problem 2-43.
5. Problem 2-45.
6. Problem 2-47.
7. Problem 2-49.
8. Problem 2-51.
9. Problem 3-19.
10. Problem 3-21.
11. Problem 3-23.
12. Problem 3-25.
13. Problem 3-27.
14. Problem 3-29.
15. Problem 4-41.
16. Problem 4-43.
17. Problem 4-45.
18. Problem 5-31.
19. Problem 7-15.
20. Problem 7-17.
21. Problem 7-19.
22. Problem 8-25.
]]></content:encoded>
      <dc:date>2018-11-22T20:17:00+00:00</dc:date>
    </item>
    <item>
      <title>Harder Intractable Reductions</title>
      <link>https://www.adamconrad.dev/blog/hard-reductions/</link>
      <description><![CDATA[We’ve already looked at intractable problems with straightforward translations. In this post, we are going to look at more creative reductions for problems where we don’t have a simple, singular translation available. But first, onto the Daily Problem:

]]></description>
      <pubDate>Thu, 22 Nov 2018 20:17:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/hard-reductions/</guid>
      <content:encoded><![CDATA[We've already looked at intractable problems with straightforward translations. In this post, we are going to look at more creative reductions for problems where we don't have a simple, singular translation available. But first, onto the Daily Problem:

> This article is part of the [Data Structures and Algorithms Series](/tag/algorithms/) following the course [The Analysis of Algorithms](https://www3.cs.stonybrook.edu/~skiena/373/). If you missed the [previous article](/blog/easy-reductions/), check that out first as well as [a copy of _The Algorithm Design Manual_](https://www.amazon.com/Algorithm-Design-Manual-Steven-Skiena/dp/1849967202).

## Answers to the previous Daily Problem

> Suppose we are given a subroutine which can solve the traveling salesman decision problem in, say, linear time. Give an efficient algorithm to find the actual TSP tour by making a polynomial number of calls to this subroutine.

First we need to find the length `l` of a shortest TSP tour. Then we just go over every edge on the graph and see if, when we remove that edge, our shortest TSP tour is still `l`. If it is, that edge wasn't part of the path so we remove the edge completely. The end result is a graph of only edges that build the `l` length of that shortest path. We only made a linear number of calls for our TSP decider since we just went over a linear number of edges in the graph.

## Two examples of harder reductions

Once again, a reduction is simply a way of turning one concept into another; _reducing_ it into a simpler/different problem. For more difficult reductions, we don't have the straightforward algorithms that we had from our previous post and will require a bit more creativity.

### 1. Integer Programming & 3-SAT

I have to be honest, the rules for integer programming seem oddly arbitrary and kind of contrived. Integer programming in the context of combinatorial optimization has to obey 4 rules:

1. Variables are integers
2. You've got some inequality constraints over those variables
3. A maximization function over some (or all) of those variables
4. An integer target you need to hit

Again, it's best to use JavaScript to provide a concrete example:

```js
const sampleIntegerProgramming = (a, b) => {
  // constraint 1
  if (a < 1) return false;
  // constraint 2
  if (b < 0) return false;
  // constraint 3
  if (a + b > 3) return false;
  const maximizationFn = b * 2;
  const target = 3;
  return maximizationFn >= target;
}
sampleIntegerProgramming(0, 0); // false - invalid constraint 1
sampleIntegerProgramming(2, 2); // false - invalid constraint 3
sampleIntegerProgramming(1, 0); // false - maximization misses target
sampleIntegerProgramming(1, 2); // true - all rules satisfied
```

As you might expect, you can reach impossible situations. If you just change your `target` to a value of `5`, there is no possible set of integer arguments that would return a value of `true`.

We can reduce 3-SAT into integer programming; given we already know 3-SAT is hard, and we want to prove integer programming is hard, our translation will start with the 3-SAT problem. There aren't as many silly rules for 3-SAT, but we can translate it pretty clearly:

1. Variables are integers, restricted to `0` (false) and `1` (true).
2. Inequality constraints, based on rule 1, are that all variables (and their negations) are between `0` and `1`. Further, to ensure truthiness for every expression, we need to make sure all variables are less than or equal to `1` when summed with its negation (which must be greater or equal to `1`).
3. The maximization function is inconsequential as we've already mapped out all of the rules for 3-SAT, so we can keep it simple and just map this function to the identity function of one of our variables.
4. The target value also doesn't matter so to make sure it never matters we set it to something universally achievable like `0`.

With these four rules for IP described in the veil of 3-SAT, we've shown our creative translation. We know every 3-SAT can solve an IP because, under these 4 rules, every clause in 3-SAT must have an answer of `1` (true). The inequality constraints sum to ensuring we just have an answer greater than or equal to `1`. With a target of `0` we satisfy our target always since no 3-SAT problem can offer a result of false (`0`).

Flipping it around to have any IP problem provide a solution to a 3-SAT problem, we can say that we know all integer variables can only be `1` and `0`. Any `1` is a `true` in the 3-SAT problem. Any `0` is a `false` in the 3-SAT problem. These assignments simply need to satisfy all expressions in a 3-SAT problem to ensure its validity.

And just like that, we prove the reduction works both ways and shows that integer programming is a NP-hard problem just like 3-SAT.

### 2. Vertex Cover & 3-SAT

We saw the definition of vertex cover in the [previous article](/blog/easy-reductions/) and it, too, can be reduced via 3-SAT.

1. The boolean variables in 3-SAT are now two vertices (one for your variable's value, one for the negated value) in your vertex cover graph connected by an edge. We'll need `n` vertices to ensure all edges are covered since no two edges can share a vertex.
2. Each expression in 3-SAT now needs three vertices (one for each variable in a 3-SAT expression). Each of those vertices form a triangle since those variables are linked in a given expression. That gives us a number of triangles equal to the number of expressions in our 3-SAT problem. Two of the three vertices in a triangle are needed to make a vertex cover, so you're looking at `2e` number of vertices where `e` is the number of expressions in your 3-SAT problem.
3. Construct a graph with `2n + 3e` number of vertices, joining the graph made in step 1 and the graph made in step 2. What this graph looks like is a wall of variables connected to their negations at their top, and an edge connecting them to their equivalent placement in their 3-SAT triangles.

And with that, we translated 3-SAT into vertex cover. Think of this like a pie chart with a legend. Each triangle makes up the segments of the pie with a different color. Each color points to a value on the legend, saying what that color represents (either the variable or the negated variable).

To show this translation is correct, we show, just like the previous example, that one problem solves the other (and vice versa).

Every 3-SAT gives a vertex cover because when we have a set of truthful assignments for our expressions, we can find those variables in the "legend" within our vertex graph. Since those assigments are true, we know we are _covering_ one of the triangles. All of the other cross edges are covered by virtue of the fact that our "pie graph" is a triangle. The triangle configuration ensures coverability in one main portion, and the truthiness of our variables ensures we connect the triangles to the legend portion from step 1.

Conversely, every vertex cover gives a 3-SAT because in a graph of size `n + 2e` (the necessary size of a vertex cover graph), we just need to account for all vertices. Per our earlier definition, we need `n` vertices to represent our legend of possible variables. These vertices can define our 3-SAT truth assignment. The remaining `2e` vertices have to be spread across each variable _and_ it's negation. In other words, we need to ensure that for a given 3-SAT problem, either you're connecting your cover to the variable used in the expression (e.g. `v1`) or its negation (e.g. `!v2`). If you have a vertex cover, you've got at least one cross edge covered per triangle (representing the 3-SAT expression). In other words, you're satisfying all expressions with a connection, or an evalution of `true` in 3-SAT for that particular expression.

That's a mouthful, but we can go either way which means we have a reduction from 3-SAT to vertex cover! And with that, our *final* daily problem:

## Onto the next Daily Problem

> Show that the following problem is NP-complete: for a dense subgraph `G` with integers `k` and `y`, does `G` contain a subgraph with exactly `k` vertices and at least `y` edges?

## More problems to practice on

For harder, more creative reduction problems, check out Problem 9-13 from the homework set.
]]></content:encoded>
      <dc:date>2018-11-22T20:17:00+00:00</dc:date>
    </item>
    <item>
      <title>Easy-ish Intractable Reductions</title>
      <link>https://www.adamconrad.dev/blog/easy-reductions/</link>
      <description><![CDATA[In the last article we covered the most elementary examples of how you can reduce one problem into another. In this article, we’re going to provide some more reductions in the context of computer science, particularly in relation to our extensive study of graphs. But first, onto the Daily Problem:

]]></description>
      <pubDate>Thu, 22 Nov 2018 20:17:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/easy-reductions/</guid>
      <content:encoded><![CDATA[In the last article we covered the most elementary examples of how you can reduce one problem into another. In this article, we're going to provide some more reductions in the context of computer science, particularly in relation to our extensive study of graphs. But first, onto the Daily Problem:

> This article is part of the [Data Structures and Algorithms Series](/tag/algorithms/) following the course [The Analysis of Algorithms](https://www3.cs.stonybrook.edu/~skiena/373/). If you missed the [previous article](/blog/reductions/), check that out first as well as [a copy of _The Algorithm Design Manual_](https://www.amazon.com/Algorithm-Design-Manual-Steven-Skiena/dp/1849967202).

## Answers to the previous Daily Problem

> We wish to compute the laziest way to dial given `n`-digit number on a standard push-button telephone using two fingers. We assume that the two fingers start out on the `*` and `#` keys, and that the effort required to move a finger from one button to another is proportional to the Euclidean distance between them. Design and analyze an algorithm that computes in time `O(n)` the method of dialing that involves moving your fingers the smallest amount of total distance.

And you thought you could get rid of dynamic programming, didn't you? So we've got 10 numbers on the keypad plus the special characters. In other words, you're asking yourself "how many phone numbers can I construct with two fingers?"

This breaks down into a bunch of subset of problems like "how many 10-digit numbers can I construct with my fingers on `*` and `#`?" which is a superset of the configuration where you slide your left finger up by 1 unit and pressed `7`, or "how many 9-digit numbers can I construct with my fingers on `7` and `#`". That is a superset of the configuration where you slide your left finger up by 1 again and pressed `4`, or "how many 8-digit numbers can I construct with my fingers on `4` and `#`." I think you see where this is going.

To drill this into your brains: the key here is that we see this repetition and _use the previous results via a cache to speed up our program_. Thus, all of those 10-digit phone numbers that start with `7`, we remember we store "move up 1" as a way to construct a phone number starting with `7`. We never have to worry about figuring that out again for all other numbers starting with `7`.

Because we're saving time by increasing the space, we reduce this algorithm down from something exponential to something linear. The algorithm, at a high level, looks like this:

1. Set a default value of `#` and `*` for your starting points. For a 1-digit or 2-digit number, the cost is simply the distance from that key to the new number for one finger (or two fingers).
2. For the remaining digits with a number range from `0` to `9`, store the cost in a 3D matrix `mtx[digit][fromLeft][fromRight]` where the first axis `digit` is the number of digits into the sequence you're in and `fromLeft` (or `fromRight`) is the successive score coming from a certain key on the keyboard given where your finger was previously placed.
3. Return the score of the matrix `mtx` for the `n` digits specified starting with the `*` and `#` keys.

Since we're filling one cell at a time in the matrix, this is just a simple run of our `n` digits across a matrix of space `100n`, since there are `10n` cells for each finger. We can reduce `O(100n)` down to `O(n)`, which is a linear-time algorithm.

## Three examples of easier reductions

To reiterate, a reduction is simply a way of turning one concept into another; _reducing_ it into a simpler/different problem. Here are a few more examples that we can extend upon from last time.

### 1. Hamiltonian Cycle & TSP

We all know the Traveling Salesman Problem: getting from one point to another as fast as possible. The Hamiltonian cycle is a similar problem with two major distinctions:

1. The graph is unweighted (i.e. in TSP every vertex has a cost, but in HC those costs are all the same)
2. The goal is to tour all elements exactly once. TSP also visits every edge (to find the best path), but that isn't the goal

Due to their similarities, we can express a Hamiltonian cycle as a reduction of a Traveling Salesman Problem:

1. First, make a copy of your Hamiltonian graph such that all vertices have the same weight
2. For every vertex against every vertex (an `n^2` loop), if the combo of these two edges expresses a vertex in your graph, set the weight of that vertex to be `1`. Otherwise, set the weight to `2`.
3. Use this newly-weighted Graph to decide if there is an answer to the TSP across all `n` of these vertices.

We know this works because if we find a Hamiltonian cycle on this graph, we can trace a TSP solution that has translated across all valid edges. Since each edge is of weight `1` and we have to explore all `n` vertices, the cost of the TSP tour has to be `n`. Reducing that back into the Hamiltonian cycle graph which has no weights, it would only use the edges of weight `1` which is the cycle we're aiming to find.

### 2. Vertex Cover & Independent Set

Still looking at graphs we see two more similar problems revolving around subsets of vertices in a graph.

A _vertex cover_ is a subset of vertices in a graph that touch all edges. Conversely, an _independent set_ is a subset of vertices in a graph that don't touch each over via an edge. For certain graphs, coloring in the vertices of vertex cover look like the inverse of what an independent set would be.

So, to handle the reduction, thus proving that both problems are just as hard to tackle, we've got a pretty straightforward algorithm:

1. Make a copy of the vertex cover graph
2. Make a copy of the number of vertices in your vertex cover subset
3. Find an answer to the independent set graph using your copied graph and vertex subset

The important thing here is we transform the inputs, not the solution. In fact, we don't even need to know what the solution is. Simply by translating the inputs into independent set problem we are able to solve for the hardness of vertex cover.

Oh, and to make things more fun: we reduced TSP into a Hamiltonian cycle, and a Hamiltonian cycle can be reduced into a vertex cover, just like an independent set.

### 3. Clique & Independent Set

For our last examples we look at cliques in graphs. Yes, like social cliques of friends, graphs can form a clique which is to say that a graph where a vertex has an edge to every other vertex in some kind of subset of vertices.

An example of a five-vertex clique would be the [pentagram](https://en.wikipedia.org/wiki/Pentagram#/media/File:Regular_star_polygon_5-2.svg) formed within a pentagon. Each corner of the pentagon forms a shape, connected the adjacent vertices. To solidify the clique, you now create edges to all of the other vertices that aren't part of the pentagon's outer shape. And what do you know, a clique is just a reduction of an independent set.

1. Copy your independent set graph, except the edges in this copy are the inverse of the edges in your independent set. In other words, if the edge `(i,j)` is not in your original graph, add that edge to your copy.
2. Find an answer to the clique graph with the same vertex subset and your new graph

As we can see here, we can imply the hardness of clique by the hardness of independent set, and thus the hardness of vertex cover. Where does this chain of hardness reductions end? That's where satisfiability comes in.

### Satisfiability: the one reduction to rule them all

NP-completeness derives all the way down to satisfiability. Any hard problem can be reduced down into a series of graph constraints like the three we saw above. Those problems all eventually reduce down to satisfiability. We can _satisfy_ a statement by saying at least 1 truth exists for every expression in a program. This is super easy to describe with JavaScript:

```js
let v1, v2;
v1 = true;  v2 = true;  // satisfied
v1 = false; v2 = false; // satisfied
v1 = true;  v2 = false; // not satisfied

const expr1 = v1 && !v2;
const expr2 = !v1 && v2;
return expr1 && expr2; // if this returns true, we're satisfied
```

In the above example we can find configurations for `v1` and `v2` that satisfy the final boolean expression. Certain problems, no matter how hard you try, cannot be satisfied:

```js
let v1, v2;
v1 = true;
v2 = true;

const expr1 = v1 && v2;
const expr2 = v1 && !v2;
const expr3 = !v1;
return expr1 && expr2 && expr3; // never satisfied
```

Go ahead and try the above code in your console. No matter how you arrange `v1` and `v2`, you will never get your return statement to return a value of `true`.

One of the elementary reductions from satisfiability (the parent to vertex cover) is 3-Satisfiability, or 3-SAT. This is just a specialized form of the above but every expression must contain _exactly 3 variables_. And since we know satisfiability is hard, we know 3-SAT is hard, too. This can extend for all numbers larger than 3, so a statement made up of four-variable expressions is called 4-SAT, a statement with all five-variable expressions is called 5-SAT, and so on.

Why does this only go up and not down in numbers?

With one variable in 1-SAT, if you set that variable to `true` you have a trivial problem. For 2-SAT, you can solve it in [linear time with depth-first search](https://en.wikipedia.org/wiki/2-satisfiability), a polynomial-time algorithm and thus not in the real of non-polynomial hard problems.

From the _n_-SAT problems derives vertex cover and all of the other previous elementary reductions, so we'll stop there.

## Onto the next Daily Problem

> Suppose we are given a subroutine which can solve the traveling salesman decision problem in, say, linear time. Give an efficient algorithm to find the actual TSP tour by making a polynomial number of calls to this subroutine.

## More problems to practice on

For easier reduction problems, check out these from the homework set:

1. Problem 9-8.
2. Problem 9-10.
2. Problem 9-11.
2. Problem 9-12.
]]></content:encoded>
      <dc:date>2018-11-22T20:17:00+00:00</dc:date>
    </item>
    <item>
      <title>Introduction to NP-Completeness and Intractable Problems</title>
      <link>https://www.adamconrad.dev/blog/reductions/</link>
      <description><![CDATA[We’re rounding the final corner in our algorithms series to focus on seemingly impossible problems. In our dynamic programming segment, we focused on taking very hard problems in exponential time and leveraged caching to get them down into polynomial time. We hinted that certain applications cannot benefit from this optimization, such as the Traveling Salesman Problem. So how can you possibly solve the most complex problems when they are unable to be solved in polynomial time?

]]></description>
      <pubDate>Thu, 22 Nov 2018 20:17:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/reductions/</guid>
      <content:encoded><![CDATA[We're rounding the final corner in our [algorithms series](/tag/algorithms/) to focus on seemingly impossible problems. In our dynamic programming segment, we focused on taking very hard problems in exponential time and leveraged caching to get them down into polynomial time. We hinted that certain applications cannot benefit from this optimization, such as the Traveling Salesman Problem. So how can you possibly solve the most complex problems when they are unable to be solved in polynomial time?

This next chunk will focus on these intractable problems to create approximations from other problems in order to make best estimates by leveraging solvable problems in other realms. This translation from one problem to another is called _reduction_, and we'll spend the next few posts creating our own reductions to satisfy these constraints. But first, onto the Daily Problem:

> This article is part of the [Data Structures and Algorithms Series](/tag/algorithms/) following the course [The Analysis of Algorithms](https://www3.cs.stonybrook.edu/~skiena/373/). If you missed the [previous article](/blog/dynamic-programming-part-3/), check that out first as well as [a copy of _The Algorithm Design Manual_](https://www.amazon.com/Algorithm-Design-Manual-Steven-Skiena/dp/1849967202).

## Answers to the previous Daily Problem

> Eggs break when dropped from great enough height. Specifically, there must be a floor `f` in any sufficiently tall building such that an egg dropped from the `f`th floor breaks, but one dropped from the `(f − 1)`st floor will not. If the egg always breaks, then `f = 1`. If the egg never breaks, then `f = n + 1`.
> You seek to find the critical floor `f` using an `n`-story building. The only operation you can perform is to drop an egg off some floor and see what happens. You start out with `k` eggs, and seek to drop eggs as few times as possible. Broken eggs cannot be reused. Let `E(k,n)` be the minimum number of egg droppings that will always suffice.
> (a) Show that `E(1,n) = n`.
> (b) Show that `E(k,n) = Θ(n^1/k)`.
> (c) Find a recurrence for `E(k,n)`. What is the running time of the dynamic program to find `E(k,n)`?

Before we completely leave dynamic programming, let's keep it fresh in our minds and think about how we can break this problem down.

### Answer to part A

To solve part A, we can intuit very simply that one egg requires starting at the bottom floor and, at worst, having to travel all of the way up each and every floor to prove what floor the egg breaks (if at all). Thus, 1 egg requires at most `n` floors.

### Answer to part B

To solve part B, we're leveraging binary search: as we reduce our eggs by `k-1`, we cut our floors in half which is a cost of `n/2^k-1`.

### Answer to part C

Finally, to solve part C, let's think of how we would solve this instinctually. If we have 100 floors and 5 eggs, it might initially make sense to just break the building up into 20 equal sections of floors. You may not get an exact match with such limited eggs but you know you'll be able to cover the building with all of your eggs at your disposal.

Diving in, let's say that section 1 (floors 1 through 20 in our example) doesn't break the egg, but section 2 (floors 21 and 40) does. We now have `k-1` eggs to explore a subproblem of this greater problem, because now we know our bounds have been reduced from 1 to 100 down to 21 to 40. You repeat this process, continuing to subdivide sections with your remaining eggs.

Generalizing this problem, we can say that, in the worst case, it will take us `n/s + s` attempts where `s` is the sections divided across the total number of floors `n`, plus the throw from the top section having the egg break at the top floor. What is the minimum section size we can use to reduce the number of hops?

Because we said min (or max) in that last question, this should harken back to [differential calculus](https://math.stackexchange.com/questions/321819/differential-problem-find-the-maximum-and-minimum-value). So if `f(s) = s + n/s`, then our derivative function is `f'(s) = 1 - n/s^2`, set to zero is `0 = 1 - n/s^2`. Rearranging this, we can say that `1 = n/s^2`, and therefore our section size is `s = sqrt(n)`.

To check for if this equation solves the minimum or the maximum, we now need to take the 2nd derivate of that equation, which is `f''(s) = 2n/s^3`. The rule states that if `s` is positive, we have `f''(s) > 0` and we have a minimum. Since you can't have negative number of sections for these floors, we now know we've reached our minimum floor section size. If that doesn't make sense, check out that Math Stackexchange link for a deeper explanation of how that differential equation works.

Going back to our original equation, since we know that a section is expressed as a fraction of `n`, we can then turn our equation into using only 1 variable, which means that `n/s + s` is now `n/sqrt(n) + sqrt(n)`, `n/sqrt(n)` is just `sqrt(n)`, so now it's `sqrt(n) + sqrt(n)` or `2sqrt(n)` is the maximum number of tries we need to find our solution.

All of that said, this means `E(k,n) = min( max(E(k,n-x), E(k-1,x-1)) + 1)`. What this is saying is that if you drop an egg and it doesn't break, we know the floors must be above where you're at and you still have your egg intact. The first function in the `max` equation is saying you would run this function with `n-x` remaining floors and the same number of `k` eggs. Conversely, if the egg breaks, we know the floors have to be below your current floor `x`, so that second `max` equation is saying you would run this function `E` with one less egg and the max floor being `x-1`. We need to take the worst-case scenario here which is why we are calling `max` over these two possibilities. We add in `+1` because you need to consider the cost of throwing the egg, which is a constant cost. Finally, for all possible `x` values to try against, we want to minimize the cost of these tries, which is why we wrap the _whole thing_ in a `min` function.

To compute the runtime of this dynamic programming algorithm, we know that `E(k, n)` runs across all `k` eggs and all floors `n` on a minimization function summing across all values of `n`, which reduces down to `O(kn^2)`.

## Reductions: turning one problem into another

If I had to summarize this section, it would be that *reductions are ways of expressing that two problems are identical.* If you can find a fast solution for problem A, it means, through transformation, you can find a fast solution for problem B.

### Examples

#### Closest pair of integers

One simple exmaple is finding the closest pair of numbers in an array. The naive implementation says find the difference across all numbers which is `O(n^2)`. The translation is to _sort the array first_ since the closest pairs will be adjacent to each other and therefore we can run the difference comparison in a linear fashion, bringing the complexity down to `O(nlog(n))`.

#### Least common subsequence

Another example we can utilize from our [dynamic programming section](/blog/2018-11-06-dynamic-programming-part-2/) involves the Longest Common Subsequence. We started that section off by talking about Edit Distance (i.e. spell checking resolves to either adding/removing/replacing/keeping a letter in a word). In many ways, LCS is just a _translation_ of Edit Distance: first we sort the numbers (again, by sorting we ensure the sequence is ever increasing), set `insertion` or `deletion` of numbers (to string together the longest sequence) to a cost of 1, set the `substitution` cost to infinity (since we don't allow substitutions in this problem), then simply subtract from our set the recursive call of the edit distance divided by two to get the lenght of the LCS. Why two? Every addition or subtraction has to happen as a pair; so the actual distance is about a "move" operation which doesn't affect the length of the array.

#### Least common multiple & greatest common divisor

Finally, a really common pairing of problems is in dealing with pairs of integers: finding their least common multiple (e.g. 24 and 36 has a LCM of 72) and greatest common divisor (e.g. for those same numbers the GCD is 12).

In fact, we can use one to derive the other: if we take two numbers, say `x` and `y`, and multiply them together and divide over their greatest common denominator, you get the least common multiple! Just like our previous two examples where sorting _translates_ a hard problem into an easier one, finding the GCD _translates_ the LCM problem from a intractable problem to a very easy one. Normally you'd need the prime factorization which cannot be done efficiently. But by deriving the LCD using the [Euclidean algorithm](https://en.wikipedia.org/wiki/Euclidean_algorithm) we can turn this into an extremely efficient `O(n)` algorithm!

We'll persue harder examples in the following posts, but for now let's move on to the next Daily Problem:

## Onto the next Daily Problem

> We wish to compute the laziest way to dial given `n`-digit number on a standard push-button telephone using two fingers. We assume that the two fingers start out on the `*` and `#` keys, and that the effort required to move a finger from one button to another is proportional to the Euclidean distance between them. Design and analyze an algorithm that computes in time `O(n)` the method of dialing that involves moving your fingers the smallest amount of total distance.

## More problems to practice on

The [last homework set for the class](http://www3.cs.stonybrook.edu/~skiena/373/hw/hw5.pdf) also covers chapter 9 on reductions as well, so here are your introductory problems into this space:

1. Problem 9-1.
2. Problem 9-2.
]]></content:encoded>
      <dc:date>2018-11-22T20:17:00+00:00</dc:date>
    </item>
    <item>
      <title>Dynamic Programming in JavaScript Part 3 - Limitations</title>
      <link>https://www.adamconrad.dev/blog/dynamic-programming-part-3/</link>
      <description><![CDATA[Last time we covered a bunch of examples utilizing dynamic programming. Again, I recommend you check out that article first before you check this one out otherwise you’re not really gaining too much. Today we will be concluding our section on dynamic programming by discussing its limitations and when not to use it as a strategy.

]]></description>
      <pubDate>Thu, 08 Nov 2018 23:30:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/dynamic-programming-part-3/</guid>
      <content:encoded><![CDATA[[Last time](/blog/dynamic-programming-part-2/) we covered a bunch of examples utilizing dynamic programming. Again, I recommend you check out that article first before you check this one out otherwise you're not really gaining too much. Today we will be concluding our section on dynamic programming by discussing its limitations and when _not_ to use it as a strategy.

> This article is part of the [Data Structures and Algorithms Series](/tag/algorithms/) following the course [The Analysis of Algorithms](https://www3.cs.stonybrook.edu/~skiena/373/). If you missed the [previous article](/blog/dynamic-programming-part-2/), check that out first as well as [a copy of _The Algorithm Design Manual_](https://www.amazon.com/Algorithm-Design-Manual-Steven-Skiena/dp/1849967202).

## Answers to the previous Daily Problem

> A certain string processing language allows the programmer to break a string into two pieces. It costs `n` units of time to break a string of `n` characters into two pieces, since this involves copying the old string. A programmer wants to break a string into many pieces, and the order in which the breaks are made can affect the total amount of time used. For example, suppose we wish to break a 20-character string after characters 3, 8, and 10. If the breaks are made in left-right order, then the first break costs 20 units of time, the second break costs 17 units of time, and the third break costs 12 units of time, for a total of 49 steps. If the breaks are made in right-left order, the first break costs 20 units of time, the second break costs 10 units of time, and the third break costs 8 units of time, for a total of only 38 steps. Give a dynamic programming algorithm that takes a list of character positions after which to break and determines the cheapest break cost in `O(n^3)` time.

As always, we're going to approach this with the three step process outlined before for solving dynamic programming problems:

1. Describe the problem recursively
2. Ensure parameters are polynomial and not exponential
3. If so, we can store partial results along the way to be efficient

Can we describe this recursively? We know we want to compute something along the lines of `breakCost(str, start, end)` which says "find the minimum break cost of string `str` starting at position `start` and ending at position `end`".

Taking a step back, we know each split costs `n`. With `n` characters we can possibly break after to exhaustively search for costs, we know that is now costing us `O(n^2)` because we need to create a partial solution matrix that fills in the cost for all possible breaks in all possible positions. Just like partitioning books or comparing words for edit distance, the cubic time comes in after we've created our matrix value and we're now calculating the cost of our cached value of our previous partition with the next break and finding the minimum cost of the two _for all positions from our start position to the end position_ in our `breakCost` function, which is also of some fraction of length `n`.

So given we've had so many examples of dynamic programming working, when doesn't it work?

## Algorithmic limitations of dynamic programming

In general we must use algorithms that are both correct and efficient. What conditions cause dynamic programming to fail under these circumstances?

### In correctness

**Dynamic programming fails correctness if the recurrence relation cannot be expressed.** Imagine we wanted to find the most _expensive_ path from point A to be B without visiting a location more than once. Can we do it with dynamic programming?

One problem is we can never enforce the _visiting more than once_ part. When you create a recurrence relation for this problem you're trying to compute a future cost for all remaining locations against your previously-stored solution up to some intermediate point. But in dynamic programming, computing that max cost could include points you've already visited. You could very easily create a cycle and an infinite loop here.

The other problem is ordering. Sure, you know you start at A and end at B, but nothing dictates which order we transition from point-to-point in between A and B - is it biggest to smallest? Left to right? Without an order we also have the opportunity to infinite loop if the cost pushes us into one direction that is cyclical.

As we saw in step 3 of our dynamic programming checklist, if we can't utilize a partial solution to help compute the state after the partial solution, the caching doesn't get us anywhere. In all of our previous examples we have an implicit ordering to evaluate things like strings or arrays. But when you have something like an undirected graph such as a network of cities on a map, you lose that order and you can't create a reliable recurrence relation.

### In efficiency

**Dynamic programming fails efficiency without order of the input object.** Again, without order, we don't have a reliable recurrence relation. Which means we can't express in faster polynomial time but the full exponential time. Since the partial results do nothing for us, our recursion is unbounded, and recursion blows up to exponential time in the worst case (see our naive fibonacci algorithm in [part 1](/blog/dynamic-programming-part-1/) for proof).

So yeah...if you have maps, networks, graphs without any sort of ordering...don't use dynamic programming.

## Concluding on Dynamic Programming

Dynamic programming is hard. You need to look at this stuff a bunch of times before it clicks. If you still need more examples, check out [this page](https://people.cs.clemson.edu/~bcdean/dp_practice/) for 10 more problems to practice. If you follow the three-step process of defining the recurrence relation, expressing the parameters in polynomial time, and storing the partial results in an array/matrix, you can pretty much solve any dynamic programming problem.

With enough examples and banging your head against the wall, these will start to become second nature (but you've got to practice!). And with that, onto the Daily Problem...

## Onto the next Daily Problem

> Eggs break when dropped from great enough height. Specifically, there must be a floor `f` in any sufficiently tall building such that an egg dropped from the `f`th floor breaks, but one dropped from the `(f − 1)`st floor will not. If the egg always breaks, then `f = 1`. If the egg never breaks, then `f = n + 1`.
> You seek to find the critical floor `f` using an `n`-story building. The only operation you can perform is to drop an egg off some floor and see what happens. You start out with `k` eggs, and seek to drop eggs as few times as possible. Broken eggs cannot be reused. Let `E(k,n)` be the minimum number of egg droppings that will always suffice.
> (a) Show that `E(1, n) = n`.
> (b) Show that `E(k,n) = Θ(n^1/k)`.
> (c) Find a recurrence for `E(k,n)`. What is the running time of the dynamic program to find `E(k, n)`?

## More problems to practice on

To wrap up dynamic programming the last problems in the chapter worth checking out from the homework set include:

1. Problem 8-18.
2. Problem 8-19.
]]></content:encoded>
      <dc:date>2018-11-08T23:30:00+00:00</dc:date>
    </item>
    <item>
      <title>Dynamic Programming in JavaScript Part 2 - Examples</title>
      <link>https://www.adamconrad.dev/blog/dynamic-programming-part-2/</link>
      <description><![CDATA[Last time we covered the basics of Dynamic Programming. I highly recommend you check out that article first before you check this one out otherwise you might be a bit lost. Today we will be continuing this discussion by revealing a few more examples of Dynamic Programming problems and how to solve them.

]]></description>
      <pubDate>Tue, 06 Nov 2018 23:30:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/dynamic-programming-part-2/</guid>
      <content:encoded><![CDATA[[Last time](/blog/dynamic-programming-part-1/) we covered the basics of Dynamic Programming. I highly recommend you check out that article first before you check this one out otherwise you might be a bit lost. Today we will be continuing this discussion by revealing a few more examples of Dynamic Programming problems and how to solve them.

> This article is part of the [Data Structures and Algorithms Series](/tag/algorithms/) following the course [The Analysis of Algorithms](https://www3.cs.stonybrook.edu/~skiena/373/). If you missed the [previous article](/blog/dynamic-programming-part-1/), check that out first as well as [a copy of _The Algorithm Design Manual_](https://www.amazon.com/Algorithm-Design-Manual-Steven-Skiena/dp/1849967202).

## Answers to the previous Daily Problem

> Suppose you are given three strings of characters: `X`, `Y` , and `Z`, where `|X| = n`, `|Y| = m`, and `|Z| = n + m`. `Z` is said to be a _shuffle_ of `X` and `Y` iff `Z` can be formed by interleaving the characters from `X` and `Y` in a way that maintains the left-to-right ordering of the characters from each string.
> (a) Show that `cchocohilaptes` is a shuffle of `chocolate` and `chips`, but `chocochilatspe` is not.
> (b) Give an efficient dynamic-programming algorithm that determines whether `Z` is a shuffle of `X` and `Y` . Hint: the values of the dynamic programming matrix you construct should be `Boolean`, not numeric.

The way I'm approaching this is thinking about how my brain would solve it before finding the recurrence relation. So to answer A:

Two passes over the string array `Z`. First pass you are picking off letters from `X` until `X` is empty (i.e. so find `c`, now `X` is `hocolate`). If `X` is empty (meaning all letters were found _in order_), start over and run through `Z` again with the same heuristic but look for `Y` instead. If both `X` and `Y` are empty, you have found the shuffle. This will run in `O(Z)` time since you pass over `Z` twice. You won't find this shuffle with `chocochilatspe` because when `chips` is down to `ps` it will find `p` and then `s` will be left and the two variables `X` and `Y` will not be empty.

Answering B is, of course, the real exercise Skiena is looking for us to undergo. Remember, determining a dynamic programming problem involves 3 steps:

1. Create a recurrence relation or recursive algorithm to describe the problem.
2. The parameters for your relation should be polynomial (i.e. recursion can take expontential time, if the number of solutions is less than exponential, that's polynomial or less possible solutions and thus cheap enough to pursue)
3. Organize your alogrithm so that you can leverage your cached partial results when you need them to save time in exchange for space.

So answer A is not recursive...or is it? I now realize that the "picking off" of the variables can be done recursively by simply providing the function `X`, `Y`, and `Z` with the first letters chopped off. It might make more sense if I just write down the recursive algorithm now:

```js
const weave = (str, z, newIdx = 0) => {
  if (!str.length) return z;
  for (let i = newIdx; i < z.length; i++) {
    if (str[0] === z[i]) {
      // storing the partial results into the arguments of the recursive function = DP!
      return weave(str.slice(1), z.slice(undefined,i).concat(z.slice(i+1)), newIdx);
    } else {
      newIdx++;
    }
  }
  return false;
};

const isShuffle = (x, y, z) => {
  return y === weave(x, z);
};
```

Before I explain the algorithm, the fact that I created a recursive solution with a defined number of arguments proves parts 1 and 2 of the dynamic programming requirements, so the real trick is to explain part 3.

The first trick I realized in writing this is that when you pick off all of the letters in `X`, if you have a shuffle, all that is left is `Y`, so we only ever need to "weave" in letters from `X` into `Z`! That means if the first word is picked off of the joint word, you can just compare the two strings for equality.

Now for the recursion. To recurse, we store the smaller versions of `X` and `Z` in the arguments of the function. To maintain order, we have to always check the first letter of `X` or else we don't maintain the rules of the shuffle. The annoying part comes with the larger word.

To store the partial results, this came down to removing the matched letter from the larger `Z` string _and_ ensuring when we start our `for` loop again within our recursed `weave` that we move along where we left off. This ensures that we only actually have to run this algorithm in `O(Z)` in one pass rather than 2 in answer A. In fact, if your `X` is solved before `Z` is empty, you already have `Y` so in the best case scenario `Z` is just `X.concat(Y)` in order. Therefore, you can actually solve this in the best-case sublinear time of `Ω(Z-Y)`.

Having fun yet? Let's keep exploring with more examples!

## Example 1: binomial coefficients

Everyone uses Fibonacci to talk about Dynamic Programming. The problem is if you only ever hear Fibonacci, your brain will train itself to remember the answer to calculating Fibonacci, and not necessarily how to apply Dynamic Programming.

Let's take another example from the book: binomial coefficients. This is just another way of saying:

> How many different ways can I count something?

So if I have the numbers `1,2,3,4` and I want to pick 2 numbers from them, how many pairs will that generate? The answer is 6, because I can grab `(1,2)`, `(1,3)`, `(1,4)`, `(2,3)`, `(2,4)`, and `(3,4)`.

A technique for quickly generating these coefficients is [Pascal's triangle](https://en.wikipedia.org/wiki/Pascal%27s_triangle) which you can read on your own for fun. Suffice it to say, it looks an awful lot like how you would generate a Fibonacci sequence. In this case, it's the sum of two numbers directly above it in the triangle.

We can map this idea out as a function for our binomial coefficient pretty quickly in code. Remember that for Fibonacci we iterated on our code three times: once with recursion, once with recursion and memoization, and finally a third with in-place memoization over a loop. To illustrate that we don't need recursion to make Dynamic Programming work, let's see binomial coefficients mapped out as a loop:

```js
const binomialCoefficient = (n, k) => {
  const bc = [[]];
  // fill along the edges of the triangle
  for (let i = 0; i < n; i++) bc[i][0] = 1;
  for (let j = 0; j < n; j++) bc[j][j] = 1;
  // fill in the middle
  for (let i = 1; i < n; i++)
    for (let j = 1; j < i; j++)
      bc[i][j] = bc[i - 1][j - 1] + bc[i - 1][j];

  return bc[n][k];
}
```

You'll see that the bigger `for` loop is just the definition of the [recurrence relation](https://en.wikipedia.org/wiki/Recurrence_relation#Binomial_coefficients) for binomial coefficients.

You can see there's a pretty similar pattern here to the last Fibonacci algorithm we drew up. First, we fill in the default values and edge cases. Next, we iterate through our numbers to precompute values in an array. Finally, we draw values from that array to grab our coefficient value.

The only real difference is that we have to supply two values now instead of one and the math of computation is slightly varied given that the recurrence relation isn't exactly the same as that of Fibonacci.

So these two examples are pretty similar. For my last trick, let's go in a completely different direction to really cement this idea.

## Example 2: Checking for spelling errors

Did you know that you can create a spellchecker with Dynamic Programming? Yeah, you can! It's the same formula: try recursion, then memoize, then optimize. So what happens when you spell something? You're dealing with 4 scenarios for the spellchecker to validate:

1. **It matched.** Each letter of `soil` is evaluated and compared to be correct.
2. **It swapped.** For `soel`, you have to swap the `e` for an `i` when you get there.
3. **It added.** For `sol`, you have to add in the `i` when you get to `l` too early.
4. **It removed.** For `soile`, you have to remove the `e` when you expected the string to end.

And all but the first scenario carries a cost since the correct string requires no changes but each of the other three requires a change.

To map this out, we have a function that takes our correct word (from something like a dictionary) and compares it to the word we typed out. To do this we're going to create a matrix of all possibilities from how to get from a letter in our target word to a letter in our compared word. I'll write it all out now then we can explain it a bit more afterwards:

```js
const compareWords = (targetWord, comparedWord) => {
  const costs = [0,0,0]; // each slot is associated with a MATCH, INSERT, or DELETE cost
  const INSERT_COST = 0;
  const DELETE_COST = 0;
  const matrix;
  let i, j, k; // counters

  // initialize the matrix 2D array
  // our first row
  for (i = 0; i < targetWord.length; i++) {
    matrix[0][i].cost = 0;
    if (i > 0) {
      matrix[0][i].parent = 'I'; // for insert cost
    } else {
      matrix[i][0].parent =  -1; // no parent since we're in the top corner
    }
  }
  // our first column
  for (j = 0; j < compareWord.length; j++) {
    matrix[j][0].cost = 0;
    if (j > 0) {
      matrix[j][0].parent = 'D'; // for deletion cost
    } else {
      matrix[j][0].parent = -1;
    }
  }

  // determine costs
  for (i = 1; i < targetWord.length; i++) {
    for (j = 1; j < compareWord.length; j++) {
      // match cost
      costs[0] = matrix[i-1][j-1].cost + (compareWord[i] === targetWord[j] ? 0 : 1);
      // insert cost
      costs[1] = matrix[i][j-1].cost + 1; // always costs 1 to insert a letter into our target word
      // delete cost
      costs[2] = matrix[i-1][j].cost + 1; // always costs 1 to delete a letter from our target word

      // assume match cost is cheapest at first
      matrix[i][j].cost = costs[0];
      matrix[i][j].parent = 'M'; // for an exact match being made
      for (k = 0; k < 3; k++) {
        if (costs[k] < matrix[i][j].cost) {
          matrix[i][j].cost = costs[k];
          matrix[i][j].parent = k === 0 ? 'M' : k === 1 ? 'I' : 'D';
        }
      }
    }
  }

  // find the final cost at the end of both words
  return matrix[targetWord.length-1][compareWord.length-1].cost;
}
```

So what is this really doing? There's really 4 parts to this solution:

### 1. Matrix initialization

We know that the edges along the rows and columns have a fixed cost at the extremes. When you compare a word to nothing, the cost scales linearly by letter as you add one more. And we know that a blank word carries no cost.

### 2. Cost calculation

Next, we compare the current position in the matrix to some previous adjacent cell. That leaves us with 3 calculations to compute:

1. **Moving up and to the left.** We assume we matched the previous cell (i.e. if we matched, both characters can advance) so all that is left to calculate is if the cell matches or not. It costs 0 for an exact match and 1 if we have to substitute one letter for the other.
2. **Moving up.** Our target character had to advance once without advancing our comparison word. In other words, we had to insert a character which carries a cost of 1 no matter what.
3. **Moving left.** Converseley, if our target character cannot advance but our comparison word did, we would need to delete a character to reduce matching costs which, again, has a price of 1 no matter what.

### 3. Optimize our path forward

Now that our costs are calculated in comparison to previous adjacent cells, we can find the minimum cost thus far. We just iterate through all of the cost enumerations and pick the cheapest one of the three. From here we can begin moving onto the next cell in the matrix.

### 4. Determine the total cost

Once we've enumerated through every cell in the cost matrix, we simply find the final cost at the very bottom of our matrix, which has compared every letter in our `targetWord` with every letter in our `compareWord`. That final cost is the cheapest cost to navigate from the target word to our comparison word.

## String comparisons solve a few problems

The beauty of this solution is that it can really work in a variety of other problems:

### Substring matching

Need to spellcheck just a word within a whole text? Initializing the first row removes the conditional to assign an insertion cost since we're not trying to build one string into the other. Then in step 4, rather than count the cost of the letter in the big text, we just find the cost at the last point in our substring rather than the whole text.

### Longest Common Subsequence (LCS)

This is one of those classic problems in Dynamic Programming: you want to find all of the shared characters between two words if you only advance character-by-character. The example in the book uses `democrat` and `republican` with a LCS of `eca` (i.e. `e` is followed by a `c` is followed by an `a` in each of the two words, and it's of length 3 which is longer than any single letter in both words and is longer than, say, `ra` which is also a subsequence in the two words).

Leveraging our existing code, if we just make the cost of a substitution in our match calculation `2` instead of `1`, the cost is disproprotionately greater than any other cost and we'd never want to swap out letters.

Why is this the solution? Think about it this way: all we ever want are matches. We want all of the other letters out of our way, which either means insertion or deletion just to get the exact sequence of something like `LETTER-e-LETTER-LETTER-c-LETTER-LETTER-a`. The least number of insertions/deletions to form an equal patter between our target and comparison words gets us our common subsequence.

## Example 3: Partitions

Imagine you have a bunch of books in a library. The head librarian has already given you all of the books organized by number of pages in the book. He then gives you a few book dividers to segment the books into sections, ideally as uniformly as possible. How would you do it?

### A naive approach

Let's say the example is super basic: 2 dividers (3 sections) with 9 books: 100 pages, 200 pages...all the way up to 900 pages. The naive way to handle this is to divide the sum of all pages of all books by the number of sections and add in your divider when your ideal division is exceeded by the next book.

So with our example we have 4,500 total pages over 9 books to divide in 3 sections, which makes each section ideally 1,500 pages. The first section creeps over 1,500 pages after book 5, so we put a divider there and it helps that we hit exactly 1,500 pages in this section. The next section creeps over after book 7, so we put the next divider there but we only have 1,300 pages in this section. We've used up all dividers, so we're left with our last section having books 8 and 9, which has a division of 1,700 pages.

This works because while 1,500 is the ideal, the last two sections have a delta of only 200 pages in either direction, so no section is worse than the other and there's no other possible configuration that would further minimize the cost from the maximum distance of the ideal.

Of course, this is a pretty ideal example and we haven't applied any dynamic programming principles to achieve this result, which means we know there is more on how to approach this. Can we leverage any of this previous knowledge to improve our algorithm?

### Using dynamic programming

The truth is we need an exhaustive search involving recursion to solve this problem. Describing this problem with recusion and storing partial results along the way gives us our dynamic programming solution. How will we manage this?

Remember that to solve the full problem, we should be able to describe a subset of the problem. In other words, if we solved the above example with only 8 books, that should give us information to help solve the total solution with 9 books. And in the other direction, that should should be based on a solution when we only have the first 7 books. What is the cost of adding this last divider?

#### Maximum sectional cost

That cost is the sum of the remaining elements. The cost we would choose is the maximum of either this current section or the largest section thus far. Why? Either we are breaking new ground and we've forced an increase in our total worst cost per section, or we have remained within the cost cap we have defined in all previous sections. This is like if our last book in our example is 1,000 pages instead of 900 - the cost is 1,800 pages which is 300 pages from our ideal instead of the previous delta which is 200 pages (i.e. 600 + 700 = 1,300 which is 200 pages off of the first section at the ideal 1,500 pages).

#### Minimum sectional distance

Now we mentioned in the previous example that while we're totalling some maximum, we want to minimize that delta from the ideal, so how do we minimize the total cost per section? Since we have `k` sections (and `k-1` dividers), we need to minimize the cost with the remainder of books if we have `k-1` sections (or `k-2` dividers). Which is to say, we need to know if the previous remainder was trending in the right direction to set us up for a reasonable maximum with our current last divider available. And because we are trying to solve this as a subset of the previous problem, _this is our recurrence relation_.

We've got enough words here and not enough code, so let's try and write out what we have above:

```js
const printBooks = (books, start, end) => {
  books.forEach(book => console.log(book));
  console.log('\n');
};

const reconstructPartition = (books, dividers, length, divisions) => {
  if (divisions === 1) return printBooks(books, 1, length);
  reconstructPartition(books, dividers, dividers[length][divisions], divisions-1);
  return printBooks(books, dividers[length][divisions]+1, length);
};

const partitionize = (books, divisions) => {
  let length = books.length;
  let allSums = [0];
  let cachedValues, dividers, cost;

  // collect the cumulative sums
  for (let i = 1; i <= length; i++) allSums[i] = allSums[i-1] + books[i];

  // boundary condition - one division is just the total sum
  for (let i = 1; i <= length; i++) cachedValues[i][1] = allSums[i];

  // boundary condition - one value across all sums is just one section
  for (let j = 1; j <= divisions; j++) cachedValues[1][j] = books[j];

  // store partial values as we walk the recurrence relation
  for (let i = 2; i <= length; i++) {
    for (let j = 2; j <= divisions; j++) {
      cachedValues[i][j] = Number.POSITIVE_INFINITY; // can't get worse than the max cost
      for (let x = 1; x <= i-1; x++) {
        cost = Math.max(cachedValues[x][j-1], allSums[i]-allSums[x]);

        if (cachedValues[i][j] > cost) {
          cachedValues[i][j] = cost;
          dividers[i][j] = x;
        }
      }
    }
  }

  return reconstructPartition(books, dividers, length, divisions);
};
```

The code here should be self-explanatory: it is meant to work out the algorithm above. The real key is iterating through all of the books and all of the divisions to get the minimum cost after finding the maximum between the previously cached value and the current sum into the partition. If we hit that, increment our dividing line and store the next cached value.

Study it a few times and if you have any questions feel free to hit me up on Twitter. We've covered a few examples here and there's a bunch to unpack so if you need to read over these a few times that's okay!

## Onto the next Daily Problem

> A certain string processing language allows the programmer to break a string into two pieces. It costs `n` units of time to break a string of `n` characters into two pieces, since this involves copying the old string. A programmer wants to break a string into many pieces, and the order in which the breaks are made can affect the total amount of time used. For example, suppose we wish to break a 20-character string after characters 3, 8, and 10. If the breaks are made in left-right order, then the first break costs 20 units of time, the second break costs 17 units of time, and the third break costs 12 units of time, for a total of 49 steps. If the breaks are made in right-left order, the first break costs 20 units of time, the second break costs 10 units of time, and the third break costs 8 units of time, for a total of only 38 steps. Give a dynamic programming algorithm that takes a list of character positions after which to break and determines the cheapest break cost in `O(n^3)` time.

## More problems to practice on

For even more dynamic programming practice problems, check out Problem 8-14 in the book.

]]></content:encoded>
      <dc:date>2018-11-06T23:30:00+00:00</dc:date>
    </item>
    <item>
      <title>Star athlete or engineer? The Big Five in tech are paying like the Big Four in sports</title>
      <link>https://www.adamconrad.dev/blog/the-big-four-sports-or-tech-companies/</link>
      <description><![CDATA[I saw this tweet about your chances of making it into the NFL so I wanted to run a similar chart of stats to see what it’s like to make it into the top echelons of the software industry.

]]></description>
      <pubDate>Sat, 03 Nov 2018 23:30:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/the-big-four-sports-or-tech-companies/</guid>
      <content:encoded><![CDATA[I saw [this tweet](https://twitter.com/GatorsScott/status/640865339320999936) about your chances of making it into the NFL so I wanted to run a similar chart of stats to see what it's like to make it into the top echelons of the software industry.

* Total people in CS: [1,256,200](https://www.bls.gov/ooh/computer-and-information-technology/software-developers.htm)
* College CS majors: [55,000](https://danwang.co/why-so-few-computer-science-majors/)
* % CS Grads who stick it out: [64%](https://datausa.io/profile/cip/110701/)
* CS grads per year: [35,329](https://datausa.io/profile/cip/110701/)
* % of CS grads to FAANG: [0.2%](https://qz.com/285001/heres-why-you-only-have-a-0-2-chance-of-getting-hired-at-google/)
* People who make it into FAANG per year: [12,500](https://qz.com/285001/heres-why-you-only-have-a-0-2-chance-of-getting-hired-at-google/)
* % of FAANG engineerins who make it to Staff level or greater: [15%](https://www.quora.com/How-does-Googles-internal-levels-work)
* People who make it to Staff engineer at Google per year: [1,875](https://www.quora.com/How-does-Googles-internal-levels-work)
* Average income of a Staff engineer at a FAANG: [$506,000](https://www.levels.fyi/SE/Google/Facebook/Amazon)
* Income after taxes: [$292,400](https://www.bankrate.com/finance/taxes/tax-brackets.aspx)

Comparing this to [professional sports starting salaries](https://cashmoneylife.com/how-much-does-a-professional-benchwarmer-earn/)

* NFL = $465,000
* MLB = $507,500
* NBA = $582,180
* NHL = $650,000

This doesn't account for the many non-rookies who spend years playing part-time or in the minor leagues, particularly in the MLB, NHL, and MLB. For example, there are a [dozen or so NBA players](https://www.basketball-reference.com/contracts/players.html) making less than your _average_ engineer in a major metro area like Silicon Valley or Boston.

What am I getting at? **A staff engineer at Google or Facebook stands to make more than the rookies in the NFL and MLB**. We are officially in an era where the top software engineers are being paid like the top sports players. What a time to be alive!


]]></content:encoded>
      <dc:date>2018-11-03T23:30:00+00:00</dc:date>
    </item>
    <item>
      <title>Dynamic Programming in JavaScript Part 1 - The Basics</title>
      <link>https://www.adamconrad.dev/blog/dynamic-programming-part-1/</link>
      <description><![CDATA[Dynamic Programming is this topic that scares the crap out of everyone (myself included). This was one of those topics that took me a super long time to think about before it clicked. But I think I figured it out! My simple way of remembering Dynamic Programming is to think of iteration with memoization.

]]></description>
      <pubDate>Thu, 01 Nov 2018 23:30:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/dynamic-programming-part-1/</guid>
      <content:encoded><![CDATA[Dynamic Programming is this topic that scares the crap out of everyone (myself included). This was one of those topics that took me a super long time to think about before it clicked. But I think I figured it out! My simple way of remembering Dynamic Programming is to think of _iteration with memoization_.

If those two things don't mean anything to you, read on, otherwise, let's recap with the previous Daily Problem as a warmup...

> This article is part of the [Data Structures and Algorithms Series](/tag/algorithms/) following the course [The Analysis of Algorithms](https://www3.cs.stonybrook.edu/~skiena/373/). If you missed the [previous article](/blog/shortest-paths/), check that out first as well as [a copy of _The Algorithm Design Manual_](https://www.amazon.com/Algorithm-Design-Manual-Steven-Skiena/dp/1849967202).

## Answers to the previous Daily Problem

> _Multisets_ are allowed to have repeated elements. A multiset of `_n_` items may thus have fewer than `_n!_` distinct permutations. For example, `{1,1,2,2}` has only six different permutations: `{1, 1, 2, 2}, {1, 2, 1, 2}, {1, 2, 2, 1}, {2, 1, 1, 2}, {2, 1, 2, 1}, and {2,2,1,1}`. Design and implement an efficient algorithm for constructing all permutations of a multiset.

This question is basically asking "give me all of the shapes of these items". Each permutation is a shape, or arrangement, of the 4 components, 2 of which are similar.

One approach would be to leverage backtracking, trying to figure out from those end results how do you get to the result with `_n-1_` values left. This is a call back to our previous articles to re-assert your knowledge of backtracking algorithms. I will offer an alternative approach below:

Another naive approach is to make an `_n^2_` loop between the number of possibilities and the number of slots and just insert each item into each slot. So 1 goes to first slot, then second slot, then third...then the second one goes into the first slot, then second slot... and you build out the possibilities by filling in the remaining options.

The problem is that doing the example above would arrive at duplicates (inserting the first 1 into the 4 slots yields the same shape of numbers as would if you inserted the second 1 into the 4 slots). So how do we eliminate these possibilities?

**Caching.** As we'll see in this article, when we cache previous results, we can throw out unnecessary moves like constructing an unnecessary permutation. Once we've inserted the four options for the first 1, we don't ever have to calculate them again for any other 1s because they yield the same output. Same with the 2s. So in the multiset example above, there are 24 different arrangements, duplicated twice (once for the 1 and once for the 2). In other words, we have `6x2x2 = 24`, or 6 permutations that are duplicated once for `1` (x2) and once again for `2` (x2).

If that doesn't make sense to you, check out how we handle generalized recurrence relations below.

## Dynamic Programming = Recursion + Memoization

Functions that recurse call themselves. _Memoization_ is just storing previously-computed results. Honestly, the best way to think about this is with a classic example, the Fibonacci Sequence:

```js
const fibonacci = (n) => {
  if (n === 1) return 0;
  if (n === 2) return 1;
  return fibonacci(n - 1) + fibonacci(n - 2);
}
```

All this algorithm says is

> To get the number you're looking for, add up the two previous numbers in the sequence

The first few numbers of Fibonacci are `0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55`. Simple, right? What's the 24th number of Fibonacci? Oof! Yeah, this one gets rough pretty quick (in fact, it is _exponentially_ bad!)

So we've got the recursive part in this function, but we're missing the memoization. If we add in memoization, this problem becomes virtually linear in time, with the tradeoff of an array to store previous results:

```js
const fibonacciCached = (n, cache = [0, 1]) => {
  if (n === 1) return 0;
  if (n === 2) return 1;
  if (!cache[n]) cache[n] = fibonacciCached(n - 1) + fibonacciCached(n - 2);
  return cache[n];
}
```

Literally one extra line to cache the result. The first solution will heat up your laptop if you find the 45th Fibonacci number from how hard it will be working. The latter will run extremely fast. All we did was make sure that if we've found this number before, we look it up in an array (constant-time lookup) as opposed to computing it all over again.

Simple, right? Well, not exactly.

For starters, this only works if you truly refer to the things you've computed. It works great here because we always refer to all of the numbers computed (since it's always referring to lower numbers). But what about backtracking a path or with depth-first search? You may not always revisit everything there, in which case you've stored things unnecessarily.

Recursion also makes things painful. Recursion really eats up stack space. If we can cache with primitives and not recurse, that's even better. With Fibonacci, it's pretty easy:

```js
const fibonacciArr = (n, cache = [0, 1]) => {
  for (i = 2; i <= n; i++) cache[i] = cache[i - 1] + cache[i - 2];
  return cache[n];
}
```

Brilliant! The shortest and the fastest solution yet with _`O(n)`_ time and space! Now, to get really weird, we have to remember one thing that you probably noticed as you were counting Fibonacci: you only ever looked back two numbers when finding the next. Could our computer do that and only have to store 2 numbers as opposed to _n_? You bet!

```js
const fibonacci2 = (n) => {
  let first = 0, second = 1, next;
  if (n <= 1) return 1;
  for (let i = 2; i < n; i++) {
    next = first + second;
    first = second;
    second = next;
  }
  return first + second;
}
```

A couple of more lines, but we all we're doing is computing the next number by adding the two previous numbers (like we first specified in our definition of Fibonacci). To save space, we overwrite the variables and keep chugging along. The solution still runs in _`O(n)`_ time but the space is now constant (the two recorded numbers plus the pointer to our upcoming to-be-computed number).

One final thing I'll say about these last two examples: you may have noticed we traded in the full caching solution for less space. If constant space and linear time is fine with you (which it probably is) then stick with this.

However, if you plan to call a function over, and over, and over again, permanently storing a singleton of the cached values, it might be better to call something like `fibonacciCached` since you can always check if someone else has done this work before. The storage space is linear while the lookup time is constant for all but the uncharted territory. If you expect to visit previous values often, consider this alternative.

## The basics are just scratching the surface

We've now introduced the fundamental aspects of Dynamic Programming with a few examples and variations. Read over the code a few times to let it sink in (trust me, it took a _really_ long time for me to see this). We've got 2 more parts of this chapter to cover, so in the meantime, take a look at the Daily Problem and we'll see you soon!

## Onto the next Daily Problem

> Suppose you are given three strings of characters: `X`, `Y` , and `Z`, where `|X| = n`, `|Y| = m`, and `|Z| = n + m`. `Z` is said to be a _shuffle_ of `X` and `Y` iff `Z` can be formed by interleaving the characters from `X` and `Y` in a way that maintains the left-to-right ordering of the characters from each string.
> (a) Show that `cchocohilaptes` is a shuffle of `chocolate` and `chips`, but `chocochilatspe` is not.
> (b) Give an efficient dynamic-programming algorithm that determines whether `Z` is a shuffle of `X` and `Y` . Hint: the values of the dynamic programming matrix you construct should be `Boolean`, not numeric.

## More problems to practice on

Now that [homework 5](http://www3.cs.stonybrook.edu/~skiena/373/hw/hw5.pdf) is out, here are a few problems that are relevant to the dynamic programming we've discussed so far:

1. Problem 8-3.
2. Problem 8-7.

Think on this and we'll check it out in the next few days as we explore more examples of Dynamic Programming.
]]></content:encoded>
      <dc:date>2018-11-01T23:30:00+00:00</dc:date>
    </item>
    <item>
      <title>Combinatorial Optimization in JavaScript</title>
      <link>https://www.adamconrad.dev/blog/combinatorial-optimization-in-js/</link>
      <description><![CDATA[If you remember from very early on in this article series we mentioned that designing algorithms requires both correctness and efficiency. For a certain set of problems, however, correctness is currently impossible.

]]></description>
      <pubDate>Tue, 30 Oct 2018 23:30:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/combinatorial-optimization-in-js/</guid>
      <content:encoded><![CDATA[If you remember from very early on in this article series we mentioned that designing algorithms requires both [correctness and efficiency](/blog/how-to-design-an-algorithm/). For a certain set of problems, however, correctness is currently _impossible_.

The reason has to do with the search space to find the answers to these problems. Permutations, for example, expand at a _factorial_ (`n!`) rate. Since modern computers can run millions of operations per second, we start to hit some difficulty when we are dealing with _as little as 13 items_ to create permutations over.

In fact, this number expands so quickly that by the time we try something as seemingly practical as finding all permutations of a deck of cards (`52!`), we're dealing with a search space [so large](https://boingboing.net/2017/03/02/how-to-imagine-52-factorial.html) that if you shuffled a deck every second _since the universe began_ you still wouldn't have seen all of the permutations!

So how do we deal with large search spaces when searching is impossible? **Enter heuristics.**

> This article is part of the [Data Structures and Algorithms Series](/tag/algorithms/) following the course [The Analysis of Algorithms](https://www3.cs.stonybrook.edu/~skiena/373/). If you missed the [previous article](/blog/maximum-flow-algorithms-networks-js/), check that out first as well as [a copy of _The Algorithm Design Manual_](https://www.amazon.com/Algorithm-Design-Manual-Steven-Skiena/dp/1849967202).

By working backwards and using heuristics to _approximate_ a correct solution, you can get close enough to a correct solution that is acceptable without having to run your computers for all of eternity. Before we get into that, let's answer the previous article's Daily Problem:

## Answers to the previous Daily Problem

> Let `G = (V,E)` be a directed weighted graph such that all the weights are positive. Let `v` and `w` be two vertices in `G` and `k ≤ |V|` be an integer. Design an algorithm to find the shortest path from `v` to `w` that contains exactly `k` edges. Note that the path need not be simple.

Keep track of all `i` hops to all `i` vertices. The shortest path for `k` edges is just the shortest path for `k-1` edges plus some edge in our `i`. We can _backtrack_ recursively from the end towards the beginning, appyling the lightest-weight edge as we head backwards towards our starting vertex.

This is the basis for how we will approach problems going forward.

## Backtracking as a means of defining heuristics

Have you ever played Sudoku? Then you can understand backtracking! Backtracking, as we alluded to earlier, is the systematic way of going through all possible configurations of a search space, whether it is how they are arranged (_permutations_) or how they are divided (_subsets_). At each step in backtracking, we are trying to solve a portion of the total problem.

Let's say we want to get from one end of the graph to the other; point A to point Z. And further we know that there are 5 different ways to get there. What backtracking will do is figure out all of the ways to get from A to Z by adding 1 more point to the solution. It will then ask "did we get to Z?" if it did, we can print out the solution, if not we have to see if we hit a dead end or if we can extend it further.

Since we're going _deeper_ into a list of possibilities leading from A to Z, it makes sense that we could implement backtracking with DFS. BFS also works since we are trying to find all possible solutions (i.e. going wide will accomplish the goal just as going deep will), but it takes up much more space since we're guaranteeing to hit all of the dead ends rather than only going deep on the ones that get us closer from A to Z.

I was going to post some code that offers a way of explaining backtracking, but to be honest after looking over the algorithm it's much easier to understand as a recipe list and then apply that to a specific problem. Therefore, let's generalize backtracking into this recipe:

1. For a given solution `A`, if it's a complete solution that gets us from A to Z, print it! If we haven't found a solution yet, move on to step 2
2. Go one level deeper - given `A`, add one more step to your partial solution. This is like going to the next fork in the road.
3. Now go down each fork in the road and proceed to step 1 with each fork as the new `A`. So if we hit a fork in the road with 3 more streets to travel down, you're re-running backtrack on `A'1`, `A'2`, and `A'3`. If we've done all of the exploring possible or hit a dead end, proceed to step 4.
4. Now that we've explored, we need to move out of where we've traveled - either to explore another road because our previous one has terminated, or to end the algorithm because we've explored all roads.

And that's the basic gist of backtracking with DFS. From here we'll look at a specific example that I mentioned earlier: Sudoku.

## Backtracking by example: Sudoku

You've heard of [Sudoku](https://en.wikipedia.org/wiki/Sudoku), right? You wouldn't guess it from the name, but this _French_ puzzle game is inherently combinatorial in nature. The object of the game is to take a 9×9 grid of blanks and fill it with the digits 1 to 9. The puzzle is completed when every row, column, and sector (3×3 subproblems corresponding to the nine sectors of a tic-tac-toe puzzle) contain the digits 1 through 9 with no deletions or repetition.

Backtracking is an excellent way to solve this problem because the game starts partially filled in. That is, Step 1 is already filled out for us (we have a given partial solution `A`) and we need to take incremental steps to get to a completed board. To begin, we need to answer 3 questions:

1. **How do we set up this problem?** Since we have a 2D board matrix, a 2D array will suit us well in JavaScript. And what do we put in these array values? The state of the board, which we hope to fill out completely by the end of the exercise.
2. **What are the possibilities at each step to evaluate?** The set of values that haven't been taken yet in a given row, column, or sector.
3. **How do we know when to backtrack?** As soon as we are out of possibilities from Question 2 (i.e. we either have filled out the board or we have an invalid board).

With all of that in mind, we can now create a custom backtracking algorithm to solve a Sudoku board!

```js
class SudokuSolver {
  const UNASSIGNED = 0;
  const DIMENSION = 9;
  const SECTOR = Math.sqrt(DIMENSION);

  constructor(board) {
    this.solve(board);
  }

  solve(board, row, col) {
    const [row, col] = this.findUnassignedLocation(row, col);

    // step 1 - print solution else find it
    if (row === -1) return this.processSolution(board);

    // step 2 - DFS; go one level deeper
    for (let number = 1; number <= DIMENSION; number++) {
      if (this.isValid(row, col, number)) {
        board[row][col] = number;

        // step 3 - if we've hit a valid spot to explore, go in!
        if (this.solve(board, row, col)) return true;

        board[row][col] = UNASSIGNED;
      }
    }

    // step 4 - we've done everything; time to backtrack!
    return false;
  }

  findUnassignedLocation(board, row, col) {
    let assigned = false;
    let currentLocation = [-1, -1];

    while (!assigned) {
      if (row === DIMENSION) {
        assigned = true;
      } else {
        if (board[row][col] === UNASSIGNED) {
          currentLocation = [row, col];
          assigned = true;
        } else {
          if (col < 8) {
            col++;
          } else {
            row++;
            col = 0;
          }
        }
      }
    }

    return currentLocation;
  }

  isValid(board, row, col, number) {
    return (
      this.isValidRow(board, row, number) &&
      this.isValidColumn(board, col, number) &&
      this.isValidSector(
        board,
        Math.floor(row / SECTOR) * SECTOR,
        Math.floor(col / SECTOR) * SECTOR,
        number
      )
    );
  }

  isValidRow(board, row, number) {
    for (var col = 0; col < DIMENSION; col++) {
      if (board[row][col] === number) return false;
    }

    return true;
  }

  isValidColumn(board, col, number) {
    for (var row = 0; row < DIMENSION; row++) {
      if (board[row][col] === number) return false;
    }

    return true;
  }

  isValidSector(board, row, col, number) {
    for (var currentRow = 0; currentRow < SECTOR; currentRow++) {
      for (var currentCol = 0; currentCol < SECTOR; currentCol++) {
        if (board[row + currentRow][col + currentCol] === number) return false;
      }
    }

    return true;
  }

  processSolution(board) {
    let boardStr = '';

    for (let i = 0; i < DIMENSION; i++) {
      for (let j = 0; j < DIMENSION; j++) {
        boardStr += grid[i][j];
      }

      boardStr += '\n';
    }

    console.log(boardStr);
  }
}
```

Alright, lots to unpack here! Let's go through the four steps one-by-one and see how we're achieving backtracking for our Sudoku solver:

### 1. Determine if the Sudoku board is solved

It may seem counterintuitive to check if you've solved the solution before you've actually gone and solved it, but we're doing this first in the algorithm because we don't want to go down any unnecessary roads. Remember, algorithms are as much about correctness as they are efficiency.

So before anything else, we need to see if we have anymore unassigned locations. That just means asking our program "are there any spots on the Sudoku board left to be filled?" If not, we process a solution (which just means printing out the items in our 2D array) otherwise we add our next step and dive in.

### 2. Loop through the numbers

Now that we have a cell, we need to see which number(s) fit into that cell. Since there are 9 numbers on a Sudoku board to work with, we iterate through all of those; and that's where the DFS part comes in.

### 3. Dive into valid configurations

Let's say we know that 3, 5, and 8 are all valid numbers that could go in our current cell. The DFS recursion basically says "let's assume we fill out our board with the 3 here, will we be able to fill out the rest of our board?" And we try to fill out that board in the universe where 3 is in this cell.

### 4. Assign or backtrack

If we reach the end state, we're done. If we reach an invalid board, then we know we have to backtrack through this whole recursive dive until we get back to where we set the 3. We then try setting the 5, and go through the whole thing again, filling out all of the remaining slots until we either reach the end state or an invalid one. Repeating with all of the valid candidates until there are no more.

And that's it! Combinatorial optimization sounds like a mouthful when taken at face value. But in reality, it's just doing a depth-first search on all possibilities until we find the right one (or ones) to solve our problem!

## Onto the next Daily Problem

Now we can apply our skills with backtracking into the Daily Problem:

> _Multisets_ are allowed to have repeated elements. A multiset of `_n_` items may thus have fewer than `_n!_` distinct permutations. For example, `{1,1,2,2}` has only six different permutations: `{1, 1, 2, 2}, {1, 2, 1, 2}, {1, 2, 2, 1}, {2, 1, 1, 2}, {2, 1, 2, 1}, and {2,2,1,1}`. Design and implement an efficient algorithm for constructing all permutations of a multiset.

## More problems to practice on

Implementing a heuristic for the [graph bandwidth problem](https://en.wikipedia.org/wiki/Graph_bandwidth) is the recommended way from the class to start diving into combinatorial search problems.

The [Eight-Queens Problem](https://medium.freecodecamp.org/lets-backtrack-and-save-some-queens-1f9ef6af5415) is another classic combinatorial backtracking algorithm to implement if you're looking for something slightly different.

Finally, the class-sponsored [Homework 4](https://www3.cs.stonybrook.edu/~skiena/373/hw/hw4.pdf) focuses purely on combinatorial search.

If you have an implementation to any of these, be sure to [share it with me on Twitter](https://twitter.com/theadamconrad)!
]]></content:encoded>
      <dc:date>2018-10-30T23:30:00+00:00</dc:date>
    </item>
    <item>
      <title>Maximum Flow Algorithms for Networks in JavaScript</title>
      <link>https://www.adamconrad.dev/blog/maximum-flow-algorithms-networks-js/</link>
      <description><![CDATA[Weighted edges present all sorts of interesting problems to solve and we took a look at a few of them. Dijkstra’s and Floyd-Warshall are great ways of approaching shortest path problems. But what if you want to find the biggest edges on purpose? And what if we aren’t just factoring in the cost of traveling down these edges, but also how fast and with how many in our caravan? Think on this while we solve the previous Daily Problem:

]]></description>
      <pubDate>Tue, 23 Oct 2018 23:30:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/maximum-flow-algorithms-networks-js/</guid>
      <content:encoded><![CDATA[Weighted edges present all sorts of interesting problems to solve and we took a look at a few of them. Dijkstra's and Floyd-Warshall are great ways of approaching shortest path problems. But what if you want to find the _biggest_ edges on purpose? And what if we aren't just factoring in the cost of traveling down these edges, but also how fast and with how many in our caravan? Think on this while we solve the previous Daily Problem:

> This article is part of the [Data Structures and Algorithms Series](/tag/algorithms/) following the course [The Analysis of Algorithms](https://www3.cs.stonybrook.edu/~skiena/373/). If you missed the [previous article](/blog/shortest-paths/), check that out first as well as [a copy of _The Algorithm Design Manual_](https://www.amazon.com/Algorithm-Design-Manual-Steven-Skiena/dp/1849967202).

## Answers to the previous Daily Problem

> Let `G = (V, E)` be an undirected weighted graph, and let `T` be the shortest-path spanning tree rooted at a vertex `v`. Suppose now that all the edge weights in `G` are increased by a constant number `k`. Is `T` still the shortest-path spanning tree from `v`?

The answer is no, and it will make sense with a simple example.

Suppose you have a triangle with two edges equal to 3 each and one edge equal to 7 (for all of your mathematicians out there, this is a contrived example so don't message me that these edge weights couldn't possibly construct a real triangle). The shortest path is going to be along the 2 3-weighted edges (with a total path weight of 6).

But if you add 10 to all 3 edges, you now have 13, 13, and 17, respectively. Now 17 is the shortest path. What was once the longest edge is now the shortest path, because if you went along the original shortest path, you'd travel 13+13 for a total path weight of 26.

## Maximum flow and bipartite matching

We mentioned in a previous article about what a _bipartite graph_ is, but at a high level, it's just a way to divide a graph into two distinct sections. For example, a set of job vertices and a set of people to do those jobs (also represented as vertices). How can we connect these two sides in a way that the maximum amount of work gets done?

This is called the _network flow problem_  because we want to be able to maximize the capacities from one side to the other so that it flows with maximum efficiency and capacity.

### Why is this important to you?

Maximum flow algorithms are often solved with something called _augmented paths_. Think of it like pouring coffee through a filter or a clogged drain. If one pipe is running slow or only has so much capacity, wouldn't you divert more water to the bigger pipe or the one that isn't clogged?

Or imagine you're a programmer on the city utilities division and you need to figure out how much water you can pump out to the town from a desalinization plant. How do you figure out where the water can go without wasting it or fully utilizing all of the available pipes? **As a front-end developer, you might have to map this relationship on a visualization.**

### Residual graphs

We can use something called a **residual graph** to figure out this network. A residual graph is like a regular graph but it also includes _flow_ and _capacity_. So if we look at a graph `G` and we layer on a residual graph `R`, we can now see, for every edge, how much flow can be pushed along an edge and, based on the weight, what the capacity that can be applied as well. With all of this extra data, we can find the inefficiencies as well as our limits.

Additionally, we can see _what direction_ we can make use of the weights of an edge. For example, if to travel from `a` to `b` along edge `(a,b)` the weight is 12, we might find out from our residual graph that from `a->b` our flow is 10 and from `b->a` our flow is only 2. Whereas another edge may reveal that regardless of direction, the price is 5. These are very key points when trying to figure out how to best utilize our network.

In fact what this tells us is that the flow on our 2nd edge is maximized, and that our primary edge of `(a,b)` has some extra capacity to utilize. How much capacity? The _minimum cut_ of that edge. Let's look at that again. The weight of `(a,b)` is 12, with one cut at 10 and one cut at 2, so **the amount we can optimize is by the minimum cut** (in this case, 2). In fact we can generalize this augmented paths problem into the algorithm represented below:

```js
class Vertex {
  constructor(
    capacity = 0,
    flow = 0,
    neighbor = null,
    nextVertex = null,
    residualCapacity = 0
  ) {
    this.capacity = capacity;
    this.flow = flow;
    this.neighbor = neighbor;
    this.nextVertex = nextVertex;
    this.residualCapacity = residualCapacity;
  }
}

class Graph {
  resetSearch() {
    for (let i = 0; i < this.vertices.length; i++) {
      PROCESSED[i] = false;
      VISITED[i] = false;
      PARENTS[i] = null;
    }
  }

  findEdge(start, end) {
    let path = this.connections[start];
    while (path) {
      if (path.adjacencyInfo === end) return path;
      path = path.nextVertex;
    }
    return path;
  }

  augmentPath(start, end, parents, volume) {
    if (start === end) return;

    let edge = this.findEdge(parents[end], end);
    edge.flow += volume;
    edge.residualCapacity -= volume;

    edge = this.findEdge(end, parents[end]);
    edge.residualCapacity += volume;

    this.augmentPath(start, parents[end], parents, volume);
  }

  pathVolume(source, sink, parents) {
    if (parents[parents.length-1] === -1) return 0;

    let edge = this.findEdge(parents[sink], sink);

    if (source === parents[sink]) {
      return edge.residualCapacity;
    } else {
      return Math.min(
        this.pathVolume(source, parents[sink], parents),
        edge.residualCapacity
      );
    }
  }

  edmondsKarp(source, sink) {
    this.addResidualEdges();
    this.resetSearch();
    this.bfs(source);

    let volume = this.pathVolume(source, sink, this.PARENTS);

    while (volume > 0) {
      this.augmentPath(source, sink, this.PARENTS, volume);
      this.resetSearch();
      this.bfs(source);
      volume = this.pathVolume(source, sink, this.PARENTS);
    }
  }
}
```

In this case, the `source` is an edge of a bipartite graph (let's say one section is `L` and the other section is `R`) and the `sink` is the edge in the other section of the graph. These edges are special because they are connected to all other edges on their side by a weight of 1. Their job is to provide easy access to all edges in the graph and map out all of the main paths and find the efficiencies. We do this via breadth-first search to look for any path from the `source` to the `sink` that increases the total flow. The algorithm, known as Edmonds-Karp, is done when we have no more extra volume left to optimize.

> Note: for the very astute, you might recognize this as the [Ford-Fulkerson method](https://en.wikipedia.org/wiki/Ford%E2%80%93Fulkerson_algorithm). Though not fully recognized as an algorithm, Edmonds-Karp is an _implementation_ of Ford-Fulkerson for maximum flow problems on networks.

Now that we've seen a bunch of algorithms for moving around a graph, here's a few things to keep in mind:

1. **Map the problem. Solve with an algorithm.** Think of these algorithms as your ace-in-the-hole. They're your ammunition for firing at problems. But you need to know what to fire at.
2. **Create the framework.** All problems can be fit into some sort of framework. Once you know the problems, the solutions are easy since they've already been given to you.
3. **Practice.** Problems are easier to recognize and slot into that framework if you see a lot of them. Make sure you hit the books and give some coding problems a try!

And with that last tip, let's get on to the Daily Problem and apply some of these recent algorithmic concepts!

## Onto the next Daily Problem

> Let `G = (V,E)` be a directed weighted graph such that all the weights are positive. Let `v` and `w` be two vertices in `G` and `k ≤ |V|` be an integer. Design an algorithm to find the shortest path from `v` to `w` that contains exactly `k` edges. Note that the path need not be simple.

## More problems to practice on

1. Problem 6-24.
2. Problem 6-25.
]]></content:encoded>
      <dc:date>2018-10-23T23:30:00+00:00</dc:date>
    </item>
    <item>
      <title>How the New York Times uses React</title>
      <link>https://www.adamconrad.dev/blog/how-the-new-york-times-uses-react/</link>
      <description><![CDATA[The New York Times is one of the largest and most respected media publications today. The Times must continue to innovate with the articles and content they write and also the digital experiences used to deliver award-winning content to serve their millions of subscribers. Over the past few years, the New York Times has invested heavily in both React and GraphQL to deliver a high-quality experience to their subscribers. This show dives into the scope and extent with which the New York Times refactored their website to incorporate React and GraphQL.

]]></description>
      <pubDate>Mon, 22 Oct 2018 10:28:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/how-the-new-york-times-uses-react/</guid>
      <content:encoded><![CDATA[The New York Times is one of the largest and most respected media publications today. The Times must continue to innovate with the articles and content they write and also the digital experiences used to deliver award-winning content to serve their millions of subscribers. Over the past few years, the New York Times has invested heavily in both React and GraphQL to deliver a high-quality experience to their subscribers. This show dives into the scope and extent with which the New York Times refactored their website to incorporate React and GraphQL.

## The Old Times

![](https://cdn-images-1.medium.com/max/2000/0*aWjIJv7-NkS_2Tcl)

Since the 1990s, the *Times* has been on a mission to digitize their archive while simultaneously expanding their audience reach by augmenting the newspaper with a digital platform. Originally the *Times* was entirely server-side rendered with PHP. Most of the pages you’re familiar with, such as the front page, individual articles, and collections are all rendered with PHP. Developers at the *Times* began to introduce React and GraphQL (first as experiments for immersive articles) along with with other open source technologies like Node and Relay in order to adapt to developer trends.

The *Times*, unlike other businesses, had immediate concerns utilizing these new open source technologies. For starters, performance is paramount. They couldn’t afford to lose readers to other major media outlets from a site that was too slow. In addition, accessibility is a major concern. A significant portion of readers listen to the *Times* with a screen reader. It is important that React be able to not only create fast pages, but also render HTML properly to support the appropriate attributes to enable accessible viewing.

## The New Times

![](https://cdn-images-1.medium.com/max/2000/0*EPygjFcOqTdwtGN8)

Regardless of the challenges, React has already been paying dividends at the *Times*. Development teams have been able to collaborate faster and with less duplications in code via reusable components on the front end. Being able to plan around components has helped many teams centralize around one central language, rather than the collection of disparate languages and repositories in the past.

The other benefit of these components is in reusability across media platforms as well. While most businesses design for desktop and maybe web, the *Times* must also account for tablets, screen readers, Apple TV, and virtually every possible device where one could read the news (and that includes legacy devices like Blackberries and Palm Pilots). Having components that can easily be transferred across devices drastically accelerates development.

React isn’t the only catalyst for development at the *Times*. With so many data sources to pull from, it can be quite cumbersome to hook all of them up in order to deliver a seamless experience for readers. GraphQL has also been a benefit because it allows one central API to read data from on the front-end, while on the back-end lots of different services and databases can connect to GraphQL to deliver that data to end-user devices. The result is clean data retrieval in React components without having to worry about how to hook up and connect to the *Times’* collection of data sources.

## Refactoring the Times with React, GraphQL, and Relay

Originally the refactor started with GraphQL. Handling the data layer was the primary consideration with so many data sources and devices consuming them. *Times* developers considered GraphQL as well as Falcor, the API server from Netflix. In evaluating these servers, the primary concern was in all of these disparate data sources: some in BigTable, some in S3, and others in APIs like Stripe. Can you grab everything in a single query? GraphQL provided the answer.

## How requests work in GraphQL

![](https://cdn-images-1.medium.com/max/2000/0*yFuUjwD2WTxTJtEH)

GraphQL works as a wrapper for requests to a server. Developers will issue JavaScript code on the client in the form of GraphQL queries. Those queries will make a request to the “server” via a normal AJAX request. That server may actually manifest as several requests. The benefit of GraphQL, Falcor, and all of these other similar services abstract away the strategies behind the hood.

For example, each of those requests may require a different strategy to handle object-relational mapping or caching of frequently requested data (whether from a CDN, in-memory stores, or relational databases).

## Integrating React with GraphQL

React can be sprinkled into your front end to handle this data retrieval via GraphQL once the queries and fragments (which are fractions of a query) are in place. A common strategy for introducing a new front-end framework is to server-side render with your existing application, whether that’s Ruby on Rails or PHP’s Laravel, and then introduce React to manage and organize your JavaScript code. With GraphQL, you can co-locate your data with your components, allowing you to easily manage the client-server relationship, making it easier to wean off of your server-side requests.

From there, you can begin to introduce GraphQL clients like Relay and Apollo. Relay works by decorating your React components to seamlessly integrate your GraphQL queries with your rendered JSX. Co-location becomes a design pattern when working with Relay.

Apollo is another GraphQL client that is framework agnostic. While both work with React, only Apollo can work with other JavaScript frameworks. Relay is also much more opinionated in terms of how you structure your application, while Apollo has a variety of options that range from minimalist integration to a complete client build.

The real benefit of a GraphQL client lies in binding the data to to your React components. While GraphQL does a great job of abstracting away all of the different data sources you want to fetch from, it doesn’t do the best job when it returns all of that varied data. That’s where these clients come in. Tools like Relay and Apollo will take in all of that data and make it easy to consume for your React components. They are meant to be the front-end frameworks for hooking up your UI to your data.

## Why GraphQL won over Falcor

![](https://cdn-images-1.medium.com/max/2048/0*R3RyK0Db8uS39Fm_)

With all of this complexity around these data aggregators, how did the *Times* ultimately decide to use GraphQL? “I think the big appealing thing about GraphQL is that it was a spec,” said James Lawrie, one of the software engineers working at the *Times*. By having a reference implementation, it meant that GraphQL was easy to integrate with all of the various languages and services that the *New York Times* utilizes.

## Why Apollo won over Relay

Relay was the early choice for the *Times* to bring data back from GraphQL to their 400+ React components. Unfortunately, limitations in flexibility and integration with React Router made it difficult for the *Times* to move forward with future versions of Relay. Because Apollo offered many of the same features as Relay, but with the additional flexibility of implementation, it made it easy for the *Times* to ultimately switch and settle on Apollo over Relay for their GraphQL client.

## Wrapping up and the future of media

![](https://cdn-images-1.medium.com/max/2048/0*zfRkIBZD1XkDISS3)

Now that the *New York Times* is fully committed to the React stack, the future is bright for future media. With React, the *Times* has additional flexibility to venture into other devices quite easily. With React Native, the *Times* can now deliver a native mobile experiences utilizing many of the same components they use on the web. The *Times* also has a 360 video and VR department, which can now take advantage of React 360, the AR/VR platform by Facebook, and the newest extension of the React ecosystem. With so many avenues to choose from, React makes it possible for the *New York Times* to stay competitive against the likes of the *Washington Post *and the *LA Times*. While the *Times* started as a print newspaper, it is now ushering in a new era of being a rich, fully-immersive digital media company, much of that thanks to React and GraphQL.

*Originally published at [softwareengineeringdaily.com](https://softwareengineeringdaily.com/2018/10/22/react-and-graphql-at-the-nytimes/) on October 22, 2018.*
]]></content:encoded>
      <dc:date>2018-10-22T10:28:00+00:00</dc:date>
    </item>
    <item>
      <title>Why you should start a newsletter even if you never grow a big audience</title>
      <link>https://www.adamconrad.dev/blog/why-you-should-start-a-newsletter-even-if-you-never-grow-a-big-audience/</link>
      <description><![CDATA[When I broke out on my own, I wanted to do everything right. One of the most common pieces of advice is to start a newsletter. I believe that even if your newsletter amounts to nothing (which it won’t), you’ll still get a great deal of value out of it.

]]></description>
      <pubDate>Sat, 20 Oct 2018 10:28:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/why-you-should-start-a-newsletter-even-if-you-never-grow-a-big-audience/</guid>
      <content:encoded><![CDATA[When I broke out on my own, I wanted to do everything right. One of the most common pieces of advice is to start a newsletter. I believe that even if your newsletter amounts to nothing (which it won’t), you’ll still get a great deal of value out of it.

## You now have a bookmark collection

I’ve tried a bunch of bookmarking services and never seemed to have stuck with them. Whether it is embedded as a feature in the browser or something like Pocket, I just can’t seem to find use in them. If I want to reference something I saw 6–12 months ago, that’s the worst. I usually can’t remember the name of the article and the search function on these services is pretty bad, so I just end up Googling for the article all over again.

Until I started a newsletter.

Now that I’m taking an** active role** pushing out links to my audience on a weekly basis, it’s helping me focus on two things: high quality articles *and* pieces that I actually read and learn from. And since these newsletter services archive my newsletters, I can quickly search and reference my e-mail blasts for the issue that contained a particular post. This is *much faster* than Googling or Pocket.

## You improve your reading, writing, and retention

Newsletters require captivating headlines, succinct writing, and expert recall to distill the content you want to share down to a sentence or a phrase. This further necessitates a command of the language as well as some mental marketing to convince your readers to click a link based on the way the content is presented.

If marketing isn’t your primary function, you’re now expanding your skill-set, becoming a better communicator, and improving your reading retention all to create content that will hook your audience into coming back for more. This can be a powerful motivator since you now have external pressure to deliver a good product and know what you’re talking about.

Now that I have to publish content regularly, I’m reading with a summary in mind (i.e. “what is the gist of this article?”) as well as any questions I might have to be a launching point for Twitter discussions, future blog posts, and more. I certainly would not be thinking with that mindset if I was passively reading articles with no end goal in mind.

## You increase your discipline through routine

Compared to a lot of tech newsletters that have been around for years, I’m still pretty new to this. But already I’ve had multiple instances where I’ve not wanted to hit that publish button to send out my newsletter. Maybe I had a vacation coming up where I wouldn’t be in the country to post on time, or I just really didn’t like the quality of articles I read in the past week. **No excuse — people are waiting for you to deliver each and every time.**

Newsletters create this social pressure to deliver content on a regular schedule, regardless of whether or not you feel like it or you’re even able to get to a computer. Even a blog can’t offer that since most publications either offer a continuous stream of content (like the bigger blog networks) or some erratic schedule that is often enough for you not to notice (i.e. any individual blogger who has established themselves).

But newsletters seem to have this expectation of routine. **People want email content delivered in a predictable manner or else they will unsubscribe.**That pressure forces you to be creative when things aren’t going perfectly.

For example, I send out my newsletters on Monday mornings, which means I have to create them by Sunday night. I’ve had plenty of Sundays so far where all I want to do is lay on the couch and watch football or I’m out of the house doing something where I won’t be near a computer in the time I need.

So instead of throwing in the towel, I get creative. I write the newsletter on a Saturday or Friday to schedule for Monday. I actively develop the newsletter in a to-do list item so that as I find articles, I write summaries *as I’m done reading them *so that when I’m ready to publish my newsletter at the end of the week, all I have to do is copy-and-paste my summaries instead of having to draft them up when my newsletter is being written each week.*

The point is **I find time to deliver which fosters discipline and willpower**. Self-help articles are oozing today with tips on how to be more productive or increase your limited vessel of willpower. Want to do all of that, and increase your reading and writing capabilities? Start a newsletter.

## Why else have you started a newsletter?

I could offer up a host of other reasons if you actually manage to grow an audience (e.g. you’ll have viewers if you ever launch a product, you’ll increase your blog traffic), but I want to make sure you understand that **you can get value from starting something like a newsletter even if it never achieves the primary goal of growing a big following.**

Are there other reasons I haven’t thought of that motivated you to start a newsletter? Were you convinced after reading this article? Throw me some claps below and let me know how mailing lists have helped you either personally or professionally.
]]></content:encoded>
      <dc:date>2018-10-20T10:28:00+00:00</dc:date>
    </item>
    <item>
      <title>Shortest Path Problems in the Real World</title>
      <link>https://www.adamconrad.dev/blog/shortest-paths/</link>
      <description><![CDATA[We’ve come along way with graphs so far. We know how to structure them, how to explore them, and how to travel them in a way that minimizes cost (however you choose to define the cost between each vertex). A special kind of cost-saving traversal deals with the shortest path between two points.

]]></description>
      <pubDate>Thu, 18 Oct 2018 10:28:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/shortest-paths/</guid>
      <content:encoded><![CDATA[We've come along way with graphs so far. We know how to structure them, how to explore them, and how to travel them in a way that minimizes cost (however you choose to define the cost between each vertex). A special kind of cost-saving traversal deals with the shortest path between two points.

Have you ever used Google Maps or Waze? If you're reading this I'm going to assume you have. **Map directions are probably the best real-world example of finding the shortest path between two points.** Today we're going to explore the algorithms for solving the shortest path problem so that you can implement your very own (vastly simplified version of) Google Maps or Waze!

> This article is part of the [Data Structures and Algorithms Series](/tag/algorithms/) following the course [The Analysis of Algorithms](https://www3.cs.stonybrook.edu/~skiena/373/). If you missed the [previous article](/blog/minimum-spanning-trees/), check that out first as well as [a copy of _The Algorithm Design Manual_](https://www.amazon.com/Algorithm-Design-Manual-Steven-Skiena/dp/1849967202).

## Answers to the previous Daily Problem

> Suppose we are given the minimum spanning tree `T` of a given graph `G` (with `n` vertices and `m` edges) and a new edge `e = (u, v)` of weight `w` that we will add to `G`. Give an efficient algorithm to find the minimum spanning tree of the graph `G + e`. Your algorithm should run in `O(n)` time to receive full credit.

Prim's or Kruskal's will suffice in solving this, but to run this in linear time we'd probably prefer Kruskal's. A graph `G + e` is no different to solve than `G` since `G` is just a subtree of `G + e`. So why should the algorithm change? Kruskal's can be used as is, but here's the distinguishing factors to look out for:

* If `e` is a backward edge, we ignore it because it's not helping us explore more of the tree. _But_ this is important because if we detect a cycle, we've now found the edge we can throw out (if it's not our current edge `e`) in our DFS exploration.
* If `e` is a forward edge, we also ignore it for the same reason, since forward edges have already been explored.
* If `e` is a tree edge, we have to compare it against all other edges leading to `v` to determine if we have to include it in our MST. If `e`'s weight `w` is less than that of the other incident weights, we will add `e` to our MST for `G + e`. Otherwise our MST hasn't changed from graph `G`.
* If `e` is a cross edge, it doesn't factor into the MST and we can safely ignore it.

This runs in `O(n)` time because our DFS to find the new edge only costs `O(n)` in a sparse graph, and once we're there it's just some constant-time operations to do comparisons to see if the new edge will be swapped into the MST or not.

## Applications for shortest paths

As we saw above, transporation problems (with solutions like Google Maps, Waze, and countless others) are a prime example of real-world applications for shortest path problems. There are a few others to consider as well if you aren't convinced yet.

### Motion planning

Have you ever used a [flip book](https://www.youtube.com/watch?v=5A0Ro4vj3KM) to animate a scene? How would you animate someone walking in that book? We all have an idea in our head as to what a human looks like when they walk. At each page of the flip book, you're using the path of the limbs to anticipate the next frame. **Flip book animation is like a shortest path problem.**

When we flip between frames in a flip book, to get to the next one, we're having our character move in the most natural (i.e. shortest) path from one point in space to the next. If you swing your leg up, it's not going to move erratically. Instead, it will move in a smooth motion, traveling along an arc that is provided in part by the contraction of your muscles and the anatomy of your bones to allow for that motion.

### Communication networks

What's the fastest way to send an email? Where's the best place to provide a CDN of your images and JavaScript? These are both shortest path problems. The answers lie in distributed networks such as Amazon Web Services and relay networks of mail servers.

This is why, for example, you are asked to choose where you want your servers to live on AWS. If you are starting a blog that caters to local businesses in Boston, it's going to be faster to serve them images and content from the `us-east` region instead of `ap-southeast`. Information travels pretty fast, but even basic spacial reasoning can convince us it will take less time to travel to our Boston customers from servers in Ohio or Virginia than from servers in Singapore or Sydney.

## How do we solve shortest path problems?

So given all of these kinds of applications, how would we go about beginning to solve them? **For unweighted graphs, BFS is sufficient.** Since all edges have equal weights, it doesn't matter how we get from A to B, just that we get there in the fewest hops, and BFS will be able to document that for us in `O(n + m)` which is linear and very efficient. But as we saw with MSTs, unweighted graphs aren't very interesting problems.

Weighted graphs are much more challenging to solve. BFS is insufficient for solving weighted graphs for shortest paths because **BFS can find _a_ short path but not the optimal shortest path.** This is because BFS could find you the path with the least weight, but requires you to traverse the most number of edges.

This is like how when you put into Google Maps for the shortest timed route, it will have you taking all of these weird shortcuts even though you know that there are more direct routes with less turns and stop signs (but probably more traffic). So how _do_ we solve the shortest path problem for weighted graphs? We're going to explore two solutions: **Dijkstra's Algorithm** and the **Floyd-Warshall Algorithm**.

### Dijkstra’s Algorithm

Dijkstra's is the premier algorithm for solving shortest path problems with weighted graphs. It's also an example of **dynamic programming**, a concept that seems to freak out many a developer.

Dynamic programming is another divide-and-conquer technique where we use the results of a subproblem in order to help answer the general problem we are trying to solve. That's it!

Dijkstra's is a dynamic programming application because if we have a path from `s->v->e` where `s` is the starting vertex and `e` is the ending one, we know that there is a middle vertex `v` such that there is a shortest path between `s->v`.

In other words, we can step back from `e` all the way back to `s` with subproblems of saying "so if we know want to know the shortest path from `s->e`, can we compute the shortest path from `s->(e-1)`? If we can find that, can we computer the shortest path from `s->(e-2)`?" and so on, until we've reached the shortest path from `s` to it's closest neighbor. _This_ is applying dynamic progamming in the form of Dijkstra's Algorithm.

```js
class Graph {
  // rest of structure from previous articles
  let ADDED, DISTANCES, PARENTS;

  dijkstra(startVertex) {
    ADDED = new Array(this.vertices.length).fill(false);
    DISTANCES = new Array(this.vertices.length).fill(Number.POSITIVE_INFINITY);
    PARENTS = new Array(this.vertices.length).fill(-1);

    DISTANCES[startVertex] = 0;
    let currentVertex = startVertex, currentEdge;

    while (!ADDED[currentVertex]) {
      ADDED[currentVertex] = true;
      currentEdge = this.connections[currentVertex];

      while (currentEdge) {
        let nextVertex = currentEdge.adjacencyInfo;
        let weight = currentEdge.weight;

        if (DISTANCES[nextVertex] > (DISTANCES[currentVertex] + weight)) {
          DISTANCES[nextVertex] = DISTANCES[currentVertex] + weight;
          PARENTS[nextVertex] = currentVertex;
        }

        currentEdge = currentEdge.nextVertex;
      }

      currentVertex = 1;
      let bestCurrentDistance = Number.POSITIVE_INFINITY;

      for (let i = 0; i < this.vertices.length; i++) {
        if (!ADDED[i] && bestCurrentDistance > DISTANCES[i]) {
          bestCurrentDistance = DISTANCES[i];
          currentVertex = i;
        }
      }
    }
  }
}
```

Does this algorithm look familiar? It should. Dijkstra's Algorithm is, in fact, _extremely_ similar to [Prim's Algorithm](/blog/minimum-spanning-trees-in-js/) from the last article. In fact, it is so similar, I only had to change 3 lines (1 of which was the name of the function). The only real difference between Prim's and Dijkstra's is in how they compare distances.

In Prim's, we check to see if the next vertex's distance is greater than the current edge weight and if it has been added yet. If it hasn't, then we set the distance to the next vertex equal to that current edge weight and make the current vertex the parent of the next.

In Dijkstra's, all we do differently is check to see if the next vertex's distance is greater than _the current edge weight PLUS the distance of the current vertex_. That summed value is what gets added to the distance array for the next vertex, and we add the current vertex to the parent of the next vertex as normal. **In sum, all we are doing extra in Dijkstra's is factoring in the new edge weight and the distance from the starting vertex to the tree vertex it is adjacent to.**

The implication here is that Dijkstra's not only finds the shortest path from `s` to `e`, it also finds the shortest paths from `s` to _all other vertices in the graph_. By having to inspect all neighbors at every given step, **Dijkstra can map all shortest routes from all vertices**. Powerful stuff, but at a cost of `O(n^2)` since every vertex is compared to every other vertex.

### Floyd-Warshall Algorithm

With most of these graph problems so far, our examples lead us to pick vertices on the outer ends of the graph, much like how we start with the root node of a tree. But what if you wanted to start in the middle? What if you wanted to know the most centrally-located vertex in a graph? In fact, the first example I could think of is Sim City.

In Sim City, the "goal" (which I put in quotes because the game is open-ended and has no real objective ending) is to create a vibrant, happy city of people, or "sims." It's essentially one condensed simulation in urban planning. You have to provide people power for their homes, roads for them to travel to work (and places to work), and all of the amenities a local municipality needs like schools, police stations, and parks. But where do you place all of this stuff to make people happy?

For many of the buildings, like police stations, they can only operate in a certain radius to effectively stop crime before it's too late. Logically, if you put a police station on the edge of town and someone commits a crime on the other end, it's going to take more time for the police cars to arrive on the scene than if it were centrally located.

And since cities in Sim City can be quite large, it's not sufficient to just place one police station in the very middle of the map and hope for the best. You'll need several stations to cover the entire map. And your map, like the real world, is not simply a square grid of flat grass and plains. Natural features like rivers, oceans, and mountains can complicate how a station can effectively police an area.

Lucky for you, there is an algorithm called **Floyd-Warshall** that can objectively find the best spot to place your buildings by finding the _all-pairs shortest path_. In other words, at every vertex we can start from we find the shortest path across the graph and see how long it takes to get to every other vertex. Each time we start over, we keep score of the total moves required for each vertex. The shorest average distance will come from that central vertex, which we can calculate with an adjacency matrix. And since we are now adding another layer of checking every vertex on top of what is essentially Dijkstra's, this algorithm runs in `O(n^3)` time.

Floyd-Warshall doesn't actually produce a singular return value of the optimal location. Instead, it returns the distance matrix with all of the optimal paths mapped out, which is often sufficient enough for most problems of this scope. Even though cubic time may seem slow, the fact is this algorithm runs fast in practice, in part because it utilizes an adjacency matrix to handle the mapping of all of its distance values (one of the rare instances that we [originally mentioned](/blog/data-structures-for-graphs/) where an adjacency matrix is a better data structure than an adjacency list). It also helps that the algorithm is simple to implement, too:

```js
class AdjacencyMatrix {
  constructor(size) {
    // ES6 gives us a nice way of filling in a 2D array
    this.weights = new Array(size).fill(
      new Array(size).fill(Number.POSITIVE_INFINITY)
    );
    this.vertices = size;
  }

  floydWarshall() {
    let distance;
    for (let k = 0; k < this.vertices; k++) {
      for (let i = 0; i < this.vertices; i++) {
        for (let j = 0; j < this.vertices; j++) {
          distance = this.weights[i][k] + this.weights[k][j];
          if (distance < this.weights[i][j]) {
            this.weight[i][j] = distance;
          }
        }
      }
    }
    return this;
  }
}
```

You can see from the triple-nested `for` loops very clearly that this is indeed a `O(n^3)` worst-case algorithm. This cost is acceptable for finding the all-pairs shortest path, but is also a good candidate for solving what are called _transitive closure_ problems. This is best explained with an example.

Who has the most power in a friend group? If mapped on a graph, you might think it's the center of the friend group, because he/she has the most immediate friends (i.e. the most direct connections to other people, or the vertex with the highest _degree_). In fact, it is the person _who has the farthest reach into the entire graph_.

The President of the United States is the most powerful person in the world not because he has the most friends, but because he has _the largest, most powerful network at his disposal_. He may not have everyone in his phone, but the people in his phone can eventually connect him to virtually anyone.

In fact, there's a popular phenomenon around this very concept of transitives closures called [Six Degrees of Kevin Bacon](https://en.wikipedia.org/wiki/Six_Degrees_of_Kevin_Bacon). It asserts that Kevin Bacon is the most powerful celebrity because "he had worked with everybody in Hollywood or someone who’s worked with them."

To prove this statement true once and for all, you could plot every Hollywood celebrity on an adjacency matrix and map their relationships with each other as edges with weights for the strength of the relationship. If Kevin Bacon has the all-pairs shortest path to every other celebrity in Hollywood then this Wikipedia entry is not just a parlor game, but a true account!

## Onto the next Daily Problem

> Let `G = (V, E)` be an undirected weighted graph, and let `T` be the shortest-path spanning tree rooted at a vertex `v`. Suppose now that all the edge weights in `G` are increased by a constant number `k`. Is `T` still the shortest-path spanning tree from `v`?

## More problems to practice on

1. Problem 6-15.
2. Problem 6-17.
]]></content:encoded>
      <dc:date>2018-10-18T10:28:00+00:00</dc:date>
    </item>
    <item>
      <title>Minimum Spanning Trees in JavaScript</title>
      <link>https://www.adamconrad.dev/blog/minimum-spanning-trees-in-js/</link>
      <description><![CDATA[Now that we’ve explored graphs and their basic algorithms, we’re going to add aonther dimension to the vertices: weight. We’ve already programmed them into our Vertex class, but now we’re going to explore how our algorithms must change when weight becomes a factor when traveling on an edge from one vertex to the next.

]]></description>
      <pubDate>Tue, 16 Oct 2018 10:28:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/minimum-spanning-trees-in-js/</guid>
      <content:encoded><![CDATA[Now that we've explored graphs and their basic algorithms, we're going to add aonther dimension to the vertices: _weight_. We've already programmed them into our `Vertex` class, but now we're going to explore how our algorithms must change when weight becomes a factor when traveling on an edge from one vertex to the next.

> This article is part of the [Data Structures and Algorithms Series](/tag/algorithms/) following the course [The Analysis of Algorithms](https://www3.cs.stonybrook.edu/~skiena/373/). If you missed the [previous article](/blog/dfs-and-topological-sort/), check that out first as well as [a copy of _The Algorithm Design Manual_](https://www.amazon.com/Algorithm-Design-Manual-Steven-Skiena/dp/1849967202).

## Answers to the previous Daily Problem

> Your job is to arrange `n` ill-behaved children in a straight line, facing front. You are given a list of `m` statements of the form _“`i` hates `j`”_. If `i` hates `j`, then you do not want put `i` somewhere behind `j`, because then `i` is capable of throwing something at `j`.

We need to order kids in a line such that they don't rip each other to shreds. Seems reasonable.

> (a) Give an algorithm that orders the line, (or says that it is not possible) in `O(m + n)` time.

So what we want is a valid list out of a graph relationship where all of the students are facing forward (hint, hint: a directed acyclic graph). Solving a DAG to produce a valid list? This sounds exactly what a topological sort would provide.

Recall that topological sort on top of DFS is just including an extra stack, and that runtime is `O(V+E)`. Here we mention `n` as the vertices and `m` as the edge relationships between any two children, so we can indeed say that our DFS topological sort will run in `O(m + n)` time.

> (b) Suppose instead you want to arrange the children in rows such that if `i` hates `j`, then `i` must be in a lower numbered row than `j`. Give an efficient algorithm to find the minimum number of rows needed, if it is possible.

Since we're dealing with rows (or going across), this lends itself to being solved with BFS instead of DFS. Tracking the depth within the tree as you traverse it, and once you find the last tree node, you now know the maximum height of the tree, which therefore indicates the number of rows that are needed.

## Definition of a minimum spanning tree

A _spanning tree_ for a graph is the set of edges that connect to all vertices in the graph. In other words, **it's the edges that make the graph fully connected**. So you might think the minimum spanning tree is the minimum set of edges that connect a graph completely. In actuality, **the minimum spanning tree (MST) is the set of edges that connect all vertices in a graph with the smallest weight possible**.

When are MSTs useful? **Any time you need to find the minimum-weighted path on a route**, such as the shortest path on a road trip (we'll learn more about shortest path problems in the next article).

### Variants

There are a few variations of spanning trees that are similar to MSTs that are also worth noting:

1. **Maximum spanning trees.** As you can imagine, this looks for the path with the heaviest edges to connect the graph instead of the lightest (I'd imagine this would be great for something like Settlers of Catan). You'll see Prim's algorithm ahead for MSTs; to handle maximum spanning trees, just negate the weights to find the heaviest/longest path.
2. **Minimum product spanning trees.** Instead of finding the edge weights with the lowest sum, you want to find the lowest product. To do this, you just add up the logarithms of the weights of the edges instead of just the weights. Why would we do this? Likely for alternative routes that have lower absolute values, sort of like how you can choose the shortest route in time as opposed tot he shortest route in distance.
3. **Minimum bottleneck spanning trees.** This just is another condition of MSTs that we ensure the maximum edge weight is minimized. We can take care of this with Kruskal's algorithm a bit later.
4. **Steiner trees.** A minimum spanning tree with [intermediate midpoints](https://en.wikipedia.org/wiki/Steiner_tree_problem). Wiring routing and circuits deal with this, so if you do any hardware engineering, you might encounter a Steiner tree.
5. **Low-degree spanning tree.** If you're visiting all vertices using a [Hamiltonian path](https://en.wikipedia.org/wiki/Hamiltonian_path) you might construct a low-degree spanning tree, but you can't solve it with the algorithms we're about to show you. The low-degree part just ensures that we aren't visiting hub nodes with a lot of outbound routes, like when you reach a rotary with a lot of exits.

### Prim's Algorithm for MSTs

Prim's algorithm is one way to find a minimum spanning tree by starting with a vertex and progressively choosing the best neighboring vertex without regard to the entire structure of the graph. When you do something like choosing the minimum edge weight for a local set of edges without regard to finding the absolute minimum to the whole structure, that is called a _greedy algorithm_.

```js
class Graph {
  // rest of structure from previous articles
  let ADDED, DISTANCES, PARENTS;

  prim(startVertex) {
    ADDED = new Array(this.vertices.length).fill(false);
    DISTANCES = new Array(this.vertices.length).fill(Number.POSITIVE_INFINITY);
    PARENTS = new Array(this.vertices.length).fill(-1);

    DISTANCES[startVertex] = 0;
    let currentVertex = startVertex, currentEdge;

    while (!ADDED[currentVertex]) {
      ADDED[currentVertex] = true;
      currentEdge = this.connections[currentVertex];

      while (currentEdge) {
        let nextVertex = currentEdge.adjacencyInfo;
        let weight = currentEdge.weight;

        if (DISTANCES[nextVertex] > weight && !ADDED[nextVertex]) {
          DISTANCES[nextVertex] = weight;
          PARENTS[nextVertex] = currentVertex;
        }

        currentEdge = currentEdge.nextVertex;
      }

      currentVertex = 1;
      let bestCurrentDistance = Number.POSITIVE_INFINITY;

      for (let i = 0; i < this.vertices.length; i++) {
        if (!ADDED[i] && bestCurrentDistance > DISTANCES[i]) {
          bestCurrentDistance = DISTANCES[i];
          currentVertex = i;
        }
      }
    }
  }
}
```

Using our Big O notation tricks for calculating the runtime, we can see that we need to iterate `n` times for all vertices which sweep across only the minimum `m` edges (which is less than or equal to `n`) to connect the whole graph, which leaves our runtime for Prim's to be `O(n^2)`. Using a priority queue can drop this time down even further to `O(m + (n log n))` (note: can you figure out why?).

### Kruskal's Algorithm for MSTs

An alternative greedy algorithm to Prim's is called Kruskal's Algorithm, which just doesn't have a starting vertex. It instead builds up connections via components of vertices and, for sparse graphs, can run faster than Prim's, provided it uses the right data structure.

```js
class Graph {
  let EDGE_PAIRS = new Array(MAX_VERTICES);

  kruskal() {
    let counter;
    let set = new SetUnion(this.vertices);

    this.toEdgeArray(); // using EDGE_PAIRS
    // sort by weight rather than a standard array value
    quicksort(this.EDGE_PAIRS, this.connections.length, this.EDGE_PAIRS.length);

    for (let i = 0; i <= this.connections; i++) {
      if (!sameComponent(set, EDGE_PAIRS[i].startVertex, EDGE_PAIRS[i].endVertex)) {
        console.log(`edge ${EDGE_PAIRS[i].startVertex},${EDGE_PAIRS[i].endVertex} in MST`);
        set.union(EDGE_PAIRS[i].startVertex, EDGE_PAIRS[i].endVertex);
      }
    }
  }
}
```

To summarize the two:

1. _Are you trying to find an MST in a sparse graph?_ **Use Kruskal's Algorithm.**
2. _Are you trying to find an MST in a dense graph?_ **Use Prim's Algorithm.**

As you can see, the data structure for fast search of a sparse graph requires something called the Union-Finding data structure, which we will discuss next.

### Union Finding

When we're dealing with MSTs, we're thinking about navigating the graph in the shortest/quickest path possible. But to find the best way around our graph, we can't just look at our immediate neighbors to determine the best way forward. It's often helpful to cluster whole "neighborhoods" of nodes together to form a better picture about where we should move next.

As we say with Kruskal's Algorithm, one way to do that is by partitioning the graph into distinct sets and sorting against that. To make that happen efficiently, we utilized a `SetUnion` data structure which we will outline below. Set unions have two primary functions:

1. `areSame()` to determine if two vertices are in the same partition
2. `merge()` to merge two partitions together

Whereas traditional nodes in a tree focus on moving down the tree and keeping track of children, set unions have nodes who focus on moving _up_ by keeping a close watch on their parents. So if you have a graph of 3 groups like so:

```
0    3    6
|   / \
1  2   4
        \
         5
```

You could represent the array of parents as `[0, 0, 3, 3, 3, 4, 6]`. A more complete example in JavaScript follows:

```js
class SetUnion {
  constructor(length) {
    this.length = length;
    this.parents = [...Array(length).keys()];
    this.elements = Array(length).fill(1);
  }

  find(element) {
    if (this.elements[element] === element) return element;
    return this.find(this.parents[element]);
  }

  areSame(set1, set2) {
    return this.find(set1) == this.find(set2);
  }

  merge(set1, set2) {
    const root1 = this.find(set1);
    const root2 = this.find(set2);

    if (root1 === root2) return "already same set";
    if (this.elements[root1] >= this.elements[root2]) {
      this.elements[root1] += this.elements[root2];
      this.parents[root2] = root1;
    } else {
      this.elements[root2] += this.elements[root1];
      this.parents[root1] = root2;
    }
  }
}
```

As you can see from above, `areSame()` and `merge()` rely on `find()` to recursively traverse the relavent subtree in the graph for the parent that matches. We know from [earlier](/blog/heapsort-priority-queues-in-js/) that recursive partitioning takes `O(log n)` time and so we can conclude that in the worst-case any set union operation will take logarithmic time, which is a great guarantee for Kruskal's Algorithm.

Now that we've explored a few algorithms for route finding, let's put some of this knowledge into practice with the Daily Problem.

## Onto the next Daily Problem

> Suppose we are given the minimum spanning tree `T` of a given graph `G` (with `n` vertices and `m` edges) and a new edge `e = (u, v)` of weight `w` that we will add to `G`. Give an efficient algorithm to find the minimum spanning tree of the graph `G + e`. Your algorithm should run in `O(n)` time to receive full credit.

## More problems to practice on

To get even more practice with graph searching and path finding, here are the other homework problems to go along with this article:

1. Implement an algorithm to print out the connected components in an undirected graph.
2. Problem 6-4.
3. Problem 6-5.
4. Problem 6-9.
]]></content:encoded>
      <dc:date>2018-10-16T10:28:00+00:00</dc:date>
    </item>
    <item>
      <title>Depth-first Search and Topological Sort in JavaScript</title>
      <link>https://www.adamconrad.dev/blog/dfs-and-topological-sort/</link>
      <description><![CDATA[Breadth-first search is a great elementary algorithm for searching graphs. In lots of scenarios, BFS will be sufficient to visit all of the vertices in a graph. In others, it’s very important that you choose the right algorith. Today we’ll be exploring depth-first search (DFS), a very similar algorithm that also explores all vertices by walking down neighboring edges, the difference is in the order in which those vertices are explored.

]]></description>
      <pubDate>Thu, 11 Oct 2018 10:28:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/dfs-and-topological-sort/</guid>
      <content:encoded><![CDATA[Breadth-first search is a great elementary algorithm for searching graphs. In lots of scenarios, BFS will be sufficient to visit all of the vertices in a graph. In others, it's very important that you choose the right algorith. Today we'll be exploring **depth-first search** (DFS), a very similar algorithm that also explores all vertices by walking down neighboring edges, the difference is in _the order in which those vertices are explored_.

First, let's tackle the Daily Problem from the previous lecture:

> This article is part of the [Data Structures and Algorithms Series](/tag/algorithms/) following the course [The Analysis of Algorithms](https://www3.cs.stonybrook.edu/~skiena/373/). If you missed the [previous article](/blog/bfs/), check that out first as well as [a copy of _The Algorithm Design Manual_](https://www.amazon.com/Algorithm-Design-Manual-Steven-Skiena/dp/1849967202).

## Answers to the previous Daily Problem

> Prove that in a breadth-first search on a undirected graph `G`, every edge is either a tree edge or a cross edge, where `x` is neither an ancestor nor descendant of `y`, in cross edge `(x,y)`.

It helps to provide definitions to the term _tree edge_ and _cross edge_. If you are traveling from `(x,y)` and it's your first time visiting `y`, we say that this is a **tree edge**. As the example mentions above, a **cross edge** is on a path between two points that is neither the ancestor nor the descendant of `x` and `y`. It also helps to define what a _forward edge_ and a _backward edge_ are to give a complete picture.

All edges adhere to one of these four types. **Tree edges** only apply to newly-visited vertices. The other three apply to previously-visited verticies. If a visit already happens for the first time, we run through this simple ruleset:

1. If the `y` in this `(x,y)` relationship is an ancestor, it is a **backward edge**.
2. If the `y` is a descendant, it is a **forward edge**.
3. If `y` is neither, that's the **cross edge**.

So let's go through all four and eliminate by contradiction.

Assume `G` has a backward edge. If a backward edge exists, there is a cycle. If we have a cycle, we have to terminate the search after an edge is processed at most once (otherwise we would loop infinitely on our cycle and we could bounce back and forth between the two vertices involved in the cycle). That means that in this case, the only other possible edges are tree edges.

Assume `G` has a forward edge. If a forward edge exists, then `(y,x)` would have already been visited during our search before we reach `x` and try to visit `y`. Since you cannot visit the descendant before you visit the ancestor, no forward edges can exist in an undirected graph.

Assume `G` has a cross edge. If a cross edge exists, we're really saying that there are connections between siblings in the tree. BFS operates by going across each layer in the height of the tree. Even though those cross edges aren't formally defined in terms of real connections, they manifest in how our output array of BFS exploration looks.

By the process of elimination, the only edge types available left are tree edges, which works just fine. Since BFS is a graph-walking algorithm, the whole point is to visit every node. By visiting a node, those edges are tree nodes their first time around. Since we don't need to visit anything a second time via our queue, our proof is satisfied.

Now that we understand a bit more about ancestors and descendants, this should make our implementation of Depth-First Search a bit clearer (Hint: DFS only has tree edges and back edges).

## Depth-First Search: Like BFS but with a stack

As I mentioned earlier, the real difference between how BFS and DFS walk a graph is in the data structure they use to make that walk. **BFS uses a queue and DFS uses a stack.** This means that there is some backtracking involved; the algorithm will go as far down the children as it can before it turns back to dive down the next stack of successive child nodes. **This tracking is what makes DFS so powerful.**

When remembering depth-first search, just remember that **we're going to visit nodes in a top-to-bottom fashion.** So a tree that looks like this:

```
      8
     / \
    6  10
   / \   \
  4   5  12
```

Will read the tree with DFS in this order: `[8, 6, 4, 5, 10, 12]`. The generic algorithm for all graphs (not just trees) using DFS looks like this:

```js
class Graph {
  // ... see last article for our implementation
  let count = 0;
  const ENTRY_TIMES = new Array(MAX_VERTICES);
  const EXIT_TIMES = new Array(MAX_VERTICES);

  dfs(currentVertex) {
    if (finished) return;
    let nextVertex;

    VISITED[currentVertex] = true;
    count += 1;
    ENTRY_TIMES[currentVertex] = count;
    console.log("PRE-PROCESSED!");
    tempVertex = this.connections[currentVertex];

    while (tempVertex) {
      nextVertex = tempVertex.adjacencyInfo;

      if (!VISITED[nextVertex]) {
        PARENTS[nextVertex] = currentVertex;
        console.log(`PROCESSED EDGE ${currentVertex}=>${nextVertex}`);
        this.dfs(nextVertex);
      } else if (
        (!PROCESSED[nextVertex] && PARENTS[currentVertex] !== nextVertex) ||
        this.isDirected
      ) {
        console.log(`PROCESSED EDGE ${currentVertex}=>${nextVertex}`);
        if (finished) return;
        tempVertex = tempVertex.nextVertex;
      }
    }

    console.log("POST-PROCESSED");
    count +=1;
    EXIT_TIMES[currentVertex] = count;
    PROCESSED[currentVertex] = true;
  }
}

```

### Why use depth-first search?

So now we know the _what_ and the _how_ of DFS, but _why_ should we care to use it? Here's a couple reasons:

* **Cycle detection.** As we mentioned from the previous Daily Problem, cycles can occur with back edges. Back edges are very easy to detect in DFS because backtracking is built into the algorithm. How long will this take? Only `O(n)` because it will take `n` vertices to find a tree node that has to head back up to an ancestor (via a back edge). Since `n` vertices need to be explored, that's only `n-1` edges to visit to find that back edge and thus the cycle. This reduces down to the worst-case `O(n)` to find the cycle in the graph.
* **Dead ends and cutoffs.** Any vertices where cutting them off disconnects the graph is called an **articulation vertex**. DFS can find these in linear time (because of the ability to look back on a parent node to see if connectivity still exists) while BFS can only do this in quadratic time.

### DFS for directed graphs: Topological sort

When graphs are directed, we now have the possibility of all for edge case types to consider. Each of these four cases helps learn more about what our graph may be doing. Recall that if no back edges exist, we have an acyclic graph. Also recall that directed acyclic graphs (DAGs) possess some interesting properties.

The most important of them is that for a certain configuration, you can represent a DAG as a list of nodes in a linear order (like you would a linked list). Such a configuration (of which more than one can exist for certain DAGs) is called a **topological sort**. Topological sorting is important because it proves that you can process any vertex before its successors. Put it another way, **we can find efficient shortest paths because only relevant vertices are involved in the topological sorting**.

```js
class Graph {
  topologicalSort() {
    for (let i = 0; i < this.vertices; i++) {
      if (!VISITED[i]) {
        // the console logs from dfs() as we pop off stack
        // will print a topological sort order
        this.dfs(i);
      }
    }
  }
}
```

## Onto the next Daily Problem

Now that we've covered the basics of graph searching, be sure to study this and the previous article to compare and contrast why these implementations work and how they're useful in different situations. Given your knowledge of both now, this next Daily Problem should give you a good idea of how to solve each part:

> Your job is to arrange `n` ill-behaved children in a straight line, facing front. You are given a list of `m` statements of the form _“`i` hates `j`”_. If `i` hates `j`, then you do not want put `i` somewhere behind `j`, because then `i` is capable of throwing something at `j`.
>
> (a) Give an algorithm that orders the line, (or says that it is not possible) in `O(m + n)` time.
>
> (b) Suppose instead you want to arrange the children in rows such that if `i` hates `j`, then `i` must be in a lower numbered row than `j`. Give an efficient algorithm to find the minimum number of rows needed, if it is possible.

## More practice problems

To wrap up this chapter here are the other homework problems to go along with these articles:

1. Problem 5-12.
2. Problem 5-13.
3. Problem 5-14.
4. Problem 5-19.
5. Problem 5-25.

Think you've got the answers? Let's see how you do in the next article!
]]></content:encoded>
      <dc:date>2018-10-11T10:28:00+00:00</dc:date>
    </item>
    <item>
      <title>An implementation of Breadth-First Search in JavaScript</title>
      <link>https://www.adamconrad.dev/blog/bfs/</link>
      <description><![CDATA[In the last article we introduced the next section in our series: graphs. Graphs are super important because essentially every major problem involved on the web can be represented as a graph. Now that we know what kinds of graphs there are and what data structures we can use to represent them, it’s time to start traversing them so we can discover every vertex and edge in a graph.

]]></description>
      <pubDate>Thu, 04 Oct 2018 10:28:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/bfs/</guid>
      <content:encoded><![CDATA[In the last article we introduced the next section in our series: graphs. Graphs are super important because essentially every major problem involved on the web can be represented as a graph. Now that we know what kinds of graphs there are and what data structures we can use to represent them, it's time to start traversing them so we can **discover every vertex and edge in a graph**.

> This article is part of the [Data Structures and Algorithms Series](/tag/algorithms/) following the course [The Analysis of Algorithms](https://www3.cs.stonybrook.edu/~skiena/373/). If you missed the [previous article](/blog/data-structures-for-graphs/), check that out first as well as [a copy of _The Algorithm Design Manual_](https://www.amazon.com/Algorithm-Design-Manual-Steven-Skiena/dp/1849967202).

## Answers to the previous Daily Problem

> Present correct and efficient algorithms to convert an undirected graph `G` between the following graph data structures. You must give the time complexity of each algorithm, assuming `n` vertices and `m` edges.

Basically we're being asked to take a graph and morph it between the data structures we learned about in the last article.

> (a) Convert from an adjacency matrix to adjacency lists.

In this case we're going from the space-heavy to the space-efficient, and mapping the relationships. In this case we iterate through every row in our 2D array, which represents each node (read: vertex) in our list. Then all we have to do is, for each node, add the `next` nodes for each pointer that is activated (read: the `1`s in the matrix). That's an `O(n^2)` algorithm since we're doing a constant amount of work for each assignment and we have to compare the `n` columns against the `n` rows.

> (b) Convert from an adjacency list to an incidence matrix. An incidence matrix `M` has a row for each vertex and a column for each edge, such that `M[i,j] = 1` if vertex `i` is part of edge `j`, `otherwise M[i,j] = 0`.

This one kind of gives it away by saying `for each` twice, since we're walking down the vertices and comparing against the other vertex on that edge giving us an `O(n+m)` algorithm.

> (c) Convert from an incidence matrix to adjacency lists.

This one is actually interesting. We know we need to iterate across all edges, so that's `O(m)` immediately. The difference is that for every `1` we find in an incidence matrix, we need to find the 2 `n` vertices that represent the two ends of that edge, therefore we need to make 2 operations for each vertex. So this conversion is actually `O(2nm)`, which reduces down to `O(nm)` because as we learned back with our article on [Big O notation](/blog/big-o-notation/), the constants on any variable are dropped.

## Graph traversal definitions

Before we dive into the various traversal algorithms, we need to introduce how we track our traversal. This is the same across all algorithms, but we want to define these states since we'll be using them quite a bit:

* **Unvisited:** You haven't visited this vertex in the graph yet.
* **Visited:** You _have_ visited this vertex.
* **Processed:** You not only visited this vertex, but you've travelled down all of its edges, too.

## Breadth-First Search

So now that we've described some definitions we'll use, let's look at our first graph traversal algorithm: **breadth-first search** (BFS for short). Later we'll look at **depth-first search**, so remove the confusion now, I want you to think on how you describe something by its _breadth_ versus its _depth_. How do you physically describe it in space?

_Breadth_ you probably think in a **left-right relationship**, like extending your arms out to show something broad and wide. The data structure that makes us think of shuffling left-to-right? **A queue.**

_Depth_ you probably think in a **up-down relationship**, like the _depths_ of the sea. The data structure that makes us think of diving down and coming back up? **A stack.**

So when remembering breadth-first search, just remember that **we're going to visit nodes in a left-to-right fashion.** So a tree that looks like this:

```
      8
     / \
    6  10
   / \   \
  4   5  12
```

Will read the tree with BFS in this order: `[8, 6, 10, 4, 5, 12]`. The generic algorithm for all graphs (not just trees) using BFS looks like this:

```js
class Graph {
  // ... see last article for our implementation

  const PROCESSED = new Array(MAX_VERTICES);
  const VISITED = new Array(MAX_VERTICES);
  const PARENTS = new Array(MAX_VERTICES);

  bfs(startVertex) {
    let visitQueue = new Queue();
    let currentVertex, nextVertex, tempVertex;

    visitQueue.enqueue(startVertex);
    visited[start] = true;

    while (visitQueue.length > 0) {
      currentVertex = visitQueue.dequeue();
      console.log("PRE-PROCESSED!");
      PROCESSED[currentVertex] = true;
      tempVertex = this.connections[currentVertex];

      while (tempVertex) {
        nextVertex = tempVertex.adjacencyInfo;

        if (!PROCESSED[nextVertex] || this.isDirected) {
          console.log(`PROCESSED EDGE ${currentVertex}=>${nextVertex}`);
        }

        if (!VISITED[nextVertex]) {
          visitQueue.enqueue(nextVertex);
          VISITED[nextVertex] = true;
          PARENTS[nextVertex] = currentVertex;
        }

        tempVertex = tempVertex.nextVertex;
      }

      console.log("POST-PROCESSED");
    }
  }
}

```

The runtime here is `O(n + m)`. Why is that? When using an adjacency list, we're going to explore all of the edges for each vertex. Every vertex will get processed once in the queue. Every edge is going to get walked on both coming and going. That makes the total number of operations `n + 2m`, so when we reduce that down for worst-case runtime, that leaves us with `O(n + m)`.

### Why use breadth-first search?

So now we know the _what_ and the _how_ of BFS, but _why_ should we care to use it? Here's a couple reasons:

* **Visualize clustered relationships.** A _cluster_ is just a subset of the graph that is connected. Remember, not all vertices need to be connected to each other. Since BFS doesn't care for an absolute root, it will pick any vertex, find all of its connections, and then keep discovering all of the explicitly-defined vertices that haven't already been discovered via connection. BFS is great for mapping people and neighbors/friends across the world. There's going to be a cluster in Australia. There's going to be a cluster in Japan. Each of these islands creates a physical cluster of relationships. They're all part of the total graph of people in the world, but they aren't all necessarily connected. The best part is, no matter how much stuff we do to find connected components, it only ever adds constant time to find clusters, so a connected clustering algorithm still runs in `O(n + m)`.
* **The Two-Color problem.** This is one of those specific graph theory math problems that's great for BFS. Each vertex gets a label (in this case, a color). The edges of said vertex cannot lead to another vertex of the same color. So `R->G->R` passes, but `R->R->B` fails. The valid graph is known as a _bipartite_ graph; _bi-_ meaning two and _partite_ meaning to divide. There are lots of real-world examples in [biology and medicine](https://academic.oup.com/gigascience/article/7/4/giy014/4875933). Have you ever played Sudoku? A Sudoku [solver](http://www.cs.kent.edu/~dragan/ST-Spring2016/SudokuGC.pdf) is just an application of graph coloring! In fact, **any sort of tree is a bipartite graph** because the whole purpose of the tree is to _divide_ the nodes involved.

## The next Daily Problem

So there's a few clear use cases for BFS. We'll find quite a few more applications for DFS in the next article, but for now, let's think on what we've learned from BFS with a Daily Problem:

> Prove that in a breadth-first search on a undirected graph `G`, every edge is either a tree edge or a cross edge, where `x` is neither an ancestor nor descendant of `y`, in cross edge `(x,y)`.

## More problems to practice on

Now that [homework 3](http://www3.cs.stonybrook.edu/~skiena/373/hw/hw3.pdf) is out, here are a few problems that are relevant to the sorting we've discussed so far:

1. Problem 5-4.
2. Problem 5-7.

That's all for today, stay tuned for depth-first search in the next article!
]]></content:encoded>
      <dc:date>2018-10-04T10:28:00+00:00</dc:date>
    </item>
    <item>
      <title>How JavaScript Engines Work</title>
      <link>https://www.adamconrad.dev/blog/how-javascript-engines-work/</link>
      <description><![CDATA[What happens when a browser loads a page? Do you know how the internals of browsers work to turn the JavaScript code you write into something usable by your computer? What happens if you want to run that same JavaScript code on your laptop instead of the browser?

]]></description>
      <pubDate>Wed, 03 Oct 2018 10:28:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/how-javascript-engines-work/</guid>
      <content:encoded><![CDATA[What happens when a browser loads a page? Do you know how the internals of browsers work to turn the JavaScript code you write into something usable by your computer? What happens if you want to run that same JavaScript code on your laptop instead of the browser?

We’ll explore what happens when your browser takes your JavaScript code and tries to run it with something called an engine.

![](https://cdn-images-1.medium.com/max/2000/0*cJHKng8sFy2y1zpJ)

Let’s walk through the journey of pulling JavaScript code from a server to rendering its output in the browser.

## Parser

![](https://cdn-images-1.medium.com/max/2000/0*x753FiiFzYUdQRDf)

When the code is successfully fetched from the server to the client, the JavaScript engine begins right away by taking that code and feeding it to the engine’s parser. A **parser** looks for patterns in the code to turn it into representative objects that the machine understands. So when the parser sees things like brackets, it knows there is a block of code to evaluate. Similarly, if it sees **var** or **let**, it knows that it should allocate space for a variable and its corresponding value. The object that stores all of the parser’s translations is known as an **[Abstract Syntax Tree](https://blog.buildo.io/a-tour-of-abstract-syntax-trees-906c0574a067)** (AST).

## Interpreter

![](https://cdn-images-1.medium.com/max/2000/0*rszNnfQgk4uoMtTs)

Next, all of the organized objects are fed from the AST to the engine’s **interpreter**, which translates those objects into **bytecode**. The bytecode step is useful because it is the first opportunity for our machine to run our code. If the interpreter contains a virtual machine (which V8 and other JavaScript engines do have) we can immediately execute code in bytecode form, without having to translate it yet again to even more highly optimized machine code.

When would be a good time to run bytecode? As Mathias Bynens (developer advocate for [Google’s V8](https://v8.dev/)) points out, bytecode is great for running one-off code. Sequences like Initialization and setup are great for this step because you can generate the bytecode, run it once, and never have to deal with it again. But what if you *do* have to keep using this code?

## Compiler

![](https://cdn-images-1.medium.com/max/2000/0*__GPaa9jvqpFEpeu)

This is where the optimization step comes in. As the interpreter is translating the AST into bytecode, it’s keeping track of all of the various idiosyncrasies of your code. Are you calling the same function over and over again with numbers? Code like this is called **hot code**, and these types of scenarios make for a good opportunity for the engine to translate that code even further into highly-optimized machine code. This machine code can be quickly accessed and executed across multiple instruction sets. This final step is run by the engine’s **compiler** to send this machine code to be run directly on the CPU.

One final thing to note: all code has to be run as machine code eventually. So even though the interpreter runs bytecode while the compiler creates optimized machine code for the CPU, the interpreter’s bytecode still gets translated into machine code. **The difference is that this interpreted machine code is *not* controlled by your engine.** Bytecode on the virtual machine will be run as it sees fit, whereas optimized compiler machine code is inspected very carefully to only run the exact instruction sets required for the CPU. Hence why this optional 3rd step exists: if we see patterns for optimization, we want to control how this machine code is run. If it doesn’t require optimization, then we’re happy to let our machine code build as it feels necessary.

How does the JavaScript engine fit into this lifecycle? **The JavaScript engine is the component of the browser that takes your JavaScript code, optimizes and executes it.** Also known as virtual machines, JavaScript engines execute your code in an environment that is platform-independent. This means that you can run the same JavaScript code on MacOS, Windows, or Linux. It also means you can run this same code in a browser, like Chrome, or on the desktop, like with Node applications such as [Electron](https://electronjs.org/).

## Differences between engines

So why are there multiple JavaScript engines if they are trying to make JavaScript code universal? Much in the way that we have multiple browser engines (Chrome, Safari, IE, Opera, Firefox, et al.), it makes sense to tie the JavaScript engine to the browser engine. By understanding how the specific browser engine works, the JavaScript engine can effectively parse, interpret, and compile the code from the browser, while also being useful enough to be the engine driving Node applications on desktop or IoT devices as well.

**The primary difference between engines lies in the choice between all of the highly-optimized compiler instructions**. Each JavaScript engine, whether it’s V8 for Chrome or[ Chakra for IE](https://github.com/Microsoft/ChakraCore), has maintainers that have to make careful choices about how to optimize for memory usage. Optimized compiler machine code is very memory-intensive, so it requires considerable trade-offs when trying to create ideal machine code. Because there is no “correct” answer, this opens up compiler optimizations to a variety of opinions and considerations, which leads to all of the various JavaScript engines we have available today.

## Considerations across devices and environments

![](https://cdn-images-1.medium.com/max/2000/0*GedPwYKJ4EvvpiPr)

As we mentioned earlier, one major advantage of using a JavaScript engine is the interpreter’s virtual machine which allows developers to create platform-agnostic JavaScript code. This code can be interpreted to run on any platform, whether it’s on the browser or on the desktop.

The browser ecosystem is considerably different from the Node ecosystem. Should your JavaScript code that runs as a server be optimized differently than JavaScript code being run as a command line tool? If so, why would an engine like V8 be the same for both optimized command line code *and* browser code?

The biggest reason against forking is **maintainability**. While it would be great to create application-specific engines that are highly optimized for specific machine interactions, the difficulty becomes maintaining an ever-increasing and sparse distribution of engines that all conform to the same [Ecma 262 standard](https://github.com/tc39/). Intuitively, it would seem much easier to just update V8 when [ES6](http://es6-features.org/) added arrow functions than to have to update V8 for Chrome, V8 for Node, and so on.

In spite of this, we still see fragmentation as a good thing, particularly for IoT devices. XS is one such example of a JavaScript engine that is specifically designed for applications that run on IoT devices. Its memory considerations are stricter than a browser-based engine like V8, allowing it to make better use of constrained devices than a typical JavaScript engine would.

## Security implications

While interoperability makes JavaScript engines a desirable complement to a browser engine, one important consideration with allowing a virtual machine to freely run foreign, untrusted code is security.

**One of the chief responsibilities of a JavaScript engine is to run untrusted code as securely as possible.** It is common for JavaScript engine teams to work with partnering teams and organizations (like [Google’s Project Zero](https://googleprojectzero.blogspot.com/)) to catch vulnerabilities and fix them in real time; sometimes even before a TC-39 spec has made it to a public release. The ability for development teams to respond to and quickly fix security vulnerabilities has allowed the development community at large to continue to rely on JavaScript engines as the primary way of executing highly performant, highly secure code across a myriad of devices and environments.

To take an even deeper dive into the world of JavaScript engines, be sure to check out Bynens’ [series on JavaScript engine fundamentals](https://mathiasbynens.be/notes/shapes-ics).

*Originally published at [softwareengineeringdaily.com](https://softwareengineeringdaily.com/2018/10/03/javascript-and-the-inner-workings-of-your-browser/) on October 3, 2018.*
]]></content:encoded>
      <dc:date>2018-10-03T10:28:00+00:00</dc:date>
    </item>
    <item>
      <title>Data structures for graphs in JavaScript</title>
      <link>https://www.adamconrad.dev/blog/data-structures-for-graphs/</link>
      <description><![CDATA[Now that we’re done with searching and sorting on basic data structures, it’s time to dive into the most applicable data structure in all of computers: graphs.

]]></description>
      <pubDate>Tue, 02 Oct 2018 10:28:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/data-structures-for-graphs/</guid>
      <content:encoded><![CDATA[Now that we're done with searching and sorting on basic data structures, it's time to dive into the most applicable data structure in all of computers: **graphs**.

> This article is part of the [Data Structures and Algorithms Series](/tag/algorithms/) following the course [The Analysis of Algorithms](https://www3.cs.stonybrook.edu/~skiena/373/). If you missed the [previous article](/blog/algorithm-behind-js-array-sort/), check that out first as well as [a copy of _The Algorithm Design Manual_](https://www.amazon.com/Algorithm-Design-Manual-Steven-Skiena/dp/1849967202).

Graphs are the grand unifier of computers, both literally and figuratively. Graphs are the most interesting problems in computer science because they reflect the real-world: networks, communication, and organization all operate in the abstract as a graph. In fact, your computer is reading this article as part of the grand graph known as the internet. Graphs are the analogy for connectivity, so when we study graphs, **we are studying how the people of the world interact with each other.**

As we mentioned in a [previous article about designing algorithms](/blog/how-to-design-an-algorithm/), when we describe graphing problems, we often use words like _network_, _circuit_, _web_, and _relationship_. When describing problems with properties of those words, the solution is often an algorithm of navigating a graph. More important than knowing solutions to graph problems is **recognizing the kinds of graphs, their data structures, and the kinds of problems they run into**.

## Kinds of graphs

There are a _lot_ of different kinds of graphs. **A graph is just a collection of nodes (or vertices) and the edges that connect them.** Mathematics succinctly defines a graph as `G = (V,E)`, so just like we use [Big O notation](/blog/big-o-notation/) to describe runtimes, so too will we use this graphing notation. But what kinds of graphs will we be dealing with?

* **By connectivity.** _Connected graphs_ have all vertices connected to each other. _Disconnected graphs_ have some vertices that have no edges connecting them. Bittorrent networks are connected graphs since each vertex offers a portion of the files they share. Bittorrent networks can also be disconnected graphs if someone has a file that isn't being shared (or desired) by any other computer on the network.
* **By direction.** _Directed graphs_ will have edges that force you to move from one vertex to the other in a certain direction. _Undirected graphs_ let you freely travel on an edge in either direction. Two-way streets are undirected. Arteries (vessels bringing blood away from the heart) and veins (vessels bringing blood back to the heart) are directed.
* **By weight.** _Weighted graphs_ put a price on the edges you travel between vertices. _Unweighted graphs_ have every edge cost the same. It costs more time (and money) to travel from Boston to Seattle than it does to travel from San Francisco to Los Angeles. It never costs anything to walk from your home to the post office and back, and that distance is always the same.
* **By simplicity.** _Simple graphs_ connect to other nodes just once. _Non-simple graphs_ can connect to other nodes with multiple edges, _or even themselves_. Tunnels, like subway systems, are simple graphs because each tunnel edge is meant to get from one place to the other. Roads, on the other hand, are non-simple because there are so many ways to get from one building to the next.
* **By density.** _Dense graphs_ have a lot of edges connecting vertices to each other. _Sparse graphs_ have only a few edges connecting vertices. Unlike the other examples, this distinction is kind of vague, but if you can describe the connections in a _quadratic_ order (i.e. every node connected to each other) it's dense. If you can describe the connections in a _linear_ order (i.e. every node is one other node or its neighbors) it's sparse. _Most problems deal with sparse graphs_. Roads are sparse graphs since each intersection is usually only the crossing of two streets. Broadcast networks are dense since a broadcast has to reach all of its subscribers.
* **By cycles.** _Cyclic graphs_ will repeat back to one origin vertex at some point. Conversely, _acyclic graphs_ do not. Tree data strutures can be described as _connected, undirected, acyclic graphs_. Adding direction to those graphs (also know as _DAGs_ for short) gives them a _topology_ (i.e. relation or arrangement) between each other. Scheduling problems where _A_ must come before _B_ are represented as DAGs and can be reasoned about with topological sorting algorithms, which we'll learn about a bit later.
* **By topology.** _Topological graphs_ have vertices that are related by how their edges are portrayed. For example, if points on a map are shaded and arranged by their distance and height like on a real topography map, that creates a topological graph. Otherwise, _embedded graphs_ represent projections of how real graphs are connected. Any drawing of a graph is an example of an embedded graph.
* **By implication.** _Implicit graphs_ have some murky details. They don't fully describe all of the connections in a graph. In fact the traversal of such a graph is how we may uncover a graph to make it explicit. _Explicit graphs_ on the other hand clearly outline all of the vertices and all of the edges from the beginning. If you've ever played Super Metroid, the [map of planet Zebes](http://www.snesmaps.com/maps/SuperMetroid/SuperMetroidMapZebes.html) starts as an implicit graph. The connection between zones is not known until you explore them. Only when you've played the whole game can you create an explicit graph of the game map.
* **By labeling.** _Labeled graphs_ give names or keys to each vertex, while _unlabeled graphs_ do not make that distinction. The fact that we can name a path between Boston and Seattle means that the graph of roads between cities (the cities being the label on each vertex) is a labeled graph.

## Data structures for graphs

Now that we know the kinds of graphs, how do we represent them in the memory of a computer? A few such data structures arise:

* **Adjacency matrix**. If there are `n` vertices in a graph, we can create an `n`x`n` matrix (i.e. an array of arrays) which represents how each vertex on a row is connected to each vertex on a column. A cell with a value of `1` means the row vertex is connected to the column vertex. Otherwise, the cell has a value of `0`.
* **Adjacency list**. A more space-efficient way to represent a sparse graph is with a linked list, where the pointers represent the edges that connect neighboring vertices to each other.

So we have two major data structures for storing the data of graph problems. How do you know when to use each?

### When adjacency matrices are a better fit

* **Testing if an edge is in a graph.** We said indexing from arrays is faster than lists. This is just a multi-dimensional application of this concept.
* **When you have really big graphs.** When space isn't a concern, indexing directly into a graph's relationships is going to be faster than having to traverse a list to find vertices deep within a graph.
* **When you're focused on writing connections (e.g. insertions and deletions of edges) to the graph.** Since we've represented every connection, it's as simple as toggling a `0` to a `1` which occurs in constant time.

### When adjacency lists are a better fit

* **Finding the degree of a vertex.** The _degree_ just means the number of connections to another vertex. Since a linked list inherently stores the pointers to other nodes, it's trivial to find the degree by simply summing the total number of pointers at a given vertex, which takes a constant-time number of operations. A matrix, however, requires summing the number of `1`s in a given row for a vertex of relationships, which involves traversing that array which takes linear time instead of constant time.
* **When you have small graphs.** Conversely to a matrix, with small graphs the relationships are simple to map as pointers in a list, which are going to be very fast to access, and much more space efficient.
* **When you're focused on reads and traveling around (i.e. traversing) a graph.** Since the data structure of a linked list bakes in relationships between nodes, this is naturally a good data structure for navigating from vertex to vertex. Matrices, on the other hand, verify relationships but offer no hint of where to go next, so it's a chaotic mess that will take at most `O(n^2)` calls on average while a list will do that in linear time.

Since most problems focus on solving these kinds of issues, **this is usually the right data structure for solving most graphing problems.**

Let's see how these might look in JavaScript:

```js
class Vertex {
  constructor(adjacencyInfo = null, nextVertex = null, weight = null) {
    this.adjacencyInfo = adjacencyInfo;
    this.nextVertex = nextVertex;
    this.weight = weight;
  }
}

class Graph {
  const MAX_VERTICES = 1000; // can be any number

  constructor(hasDirection) {
    this.connections = new Array(MAX_VERTICES).fill(null);
    this.degrees = new Array(MAX_VERTICES).fill(0);
    this.edges = 0;
    this.isDirected = hasDirection;
    this.vertices = 0;
  }

  insert(vertexStart, vertexEnd) {
    let vertex = new Vertex(vertexEnd, this.connections[vertexStart]);
    this.connections[vertexStart] = vertex;
    this.degrees[vertexStart] += 1;

    if (this.isDirected) {
      this.edges += 1;
    } else {
      this.insert(vertexEnd, vertexStart);
    }
  }

  print() {
    for (let i = 0; i < this.vertices; i++) {
      console.log(`${i}: `);
      let connection = this.connections[i];

      while (connection) {
        console.log(` ${connection.adjacencyInfo}`);
        connection = connection.next;
      }

      console.log('\n'); // new line for clarity
    }
  }
}
```

## The next Daily Problem

To get us thinking a bit about graphs, our next Daily Problem is Problem 5-8 from _The Algorithm Design Manual_:

> Present correct and efficient algorithms to convert an undirected graph `G` between the following graph data structures. You must give the time complexity of each algorithm, assuming `n` vertices and `m` edges.
>
> (a) Convert from an adjacency matrix to adjacency lists.
>
> (b) Convert from an adjacency list to an incidence matrix. An incidence matrix `M` has a row for each vertex and a column for each edge, such that `M[i,j] = 1` if vertex `i` is part of edge `j`, `otherwise M[i,j] = 0`.
>
> (c) Convert from an incidence matrix to adjacency lists.

Think you have it? Have questions? Send it over [on Twitter](https://twitter.com/theadamconrad) and I'd be happy to help, and good luck!
]]></content:encoded>
      <dc:date>2018-10-02T10:28:00+00:00</dc:date>
    </item>
    <item>
      <title>Verifying request signatures in Elixir/Phoenix</title>
      <link>https://www.adamconrad.dev/blog/verifying-request-signatures-in-elixir-phoenix/</link>
      <description><![CDATA[If you’re working with public APIs a common security task is to verify request signatures to prove the request is coming from who you believe you are really talking to.

]]></description>
      <pubDate>Sun, 30 Sep 2018 09:10:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/verifying-request-signatures-in-elixir-phoenix/</guid>
      <content:encoded><![CDATA[If you're working with public APIs a common security task is to verify request signatures to prove the request is coming from who you believe you are really talking to.

I'm working on a Slack bot that requires [verifying requests](https://api.slack.com/docs/verifying-requests-from-slack) using a custom header and a request signature. The documentation is actually pretty good, even providing a sample walkthrough in pseudocode (that leans heavily on Python).

The problem? This works in a _much_ different way with Elixir/Phoenix.

## A step-by-step walkthrough of verifying secrets in Slack

### 1. Grab your signing secret and request body

The [signing secret](https://api.slack.com/docs/verifying-requests-from-slack#signing_secrets_admin_page) is a secret token that you'd best store in your environment variables (e.g. `System.get_env("SLACK_SIGNING_SECRET")`).

The request body is the first wrench in your plan. **The raw request body is not accessible from the Phoenix `conn` at all.** Even though `params` (which includes the `body_params` from the request body) _has_ the request body, Phoenix, in a defensible move, opted to transform raw request data into an Elixir map.

So how do you grab the raw body? **You have to intercept your endpoint _before_ the standard `Plug.Parser` gets to it with your own custom parser.**

The answer is succinctly defined [within the official `Plug` docs](https://hexdocs.pm/plug/Plug.Parsers.html#module-custom-body-reader). For Phoenix, you'll need to add your request body module to your `lib/` folder and call it within your `endpoint.ex` file above your other `Plug.Parser`s to properly intercept the request body:


```elixir
# lib/cache_body_reader.ex
defmodule CacheBodyReader do
  def read_body(conn, opts) do
    {:ok, body, conn} = Plug.Conn.read_body(conn, opts)
    conn = update_in(conn.assigns[:raw_body], &[body | (&1 || [])])
    {:ok, body, conn}
  end
end

# lib/APPNAME_web/endpoint.ex
defmodule APPNAMEWeb.Endpoint do
  use Phoenix.Endpoint, otp_app: :APPNAME

  # ...

  # Your new Plug.Parser
  plug(
    Plug.Parsers,
    parsers: [:urlencoded, :json],
    pass: ["text/*"],
    body_reader: {APPNAME.CacheBodyReader, :read_body, []},
    json_decoder: Poison
  )

  # All other existing Plug.Parsers
  plug(
    Plug.Parsers,
    parsers: [:urlencoded, :multipart, :json],
    pass: ["*/*"],
    json_decoder: Poison
  )

  # ...
end
```

Now your `conn` plugs will contain a newly assigned property, `assigns[:raw_body]` which provides the stringified, raw request body.

### 2. Extract the timestamp header from the request

Getting the request timestamp is significantly easier than the raw request body, but there are still a few things that don't exactly line up with the Slack tutorial:

1. **The request headers all come into Phoenix in lowercase**
2. **Elixir timestamps are not the standard UNIX timestamp**

The request timestamp is easy to convert: grab the header from the `conn` object (returned as an `Enum`) in all lowercase and convert it to a number (headers arrive over as a string):

```elixir
timestamp =
  conn
  |> get_req_header("x-slack-request-timestamp")
  |> Enum.at(0)
  |> String.to_integer()
```

The local timestamp is kind of weird. The easiest way is to leverage the Erlang API but the numbers start with the start of the Gregorian calendar (so roughly 2018 years ago) rather than the standard UNIX timestamp (Jan 1, 1970). So you'll just have to subtract that difference, which is an oddly specific magic number:

```elixir
@unix_gregorian_offset 62_167_219_200

gregorian_timestamp =
  :calendar.local_time()
  |> :calendar.datetime_to_gregorian_seconds()

local_timestamp = gregorian_timestamp - @unix_gregorian_offset
```

Now you just need to take the absolute value of the difference and make sure it's within some reasonable delta (in the case of the tutorial, it's 5 minutes or 300 seconds):

```elixir
if abs(local_timestamp - timestamp) > 300 do
  # nothing / return false
else
  # process request / return true
end
```

### 3. Concatenate the signature string

This is the easiest step because interpolated strings are just as easy as you think they are, and now that you have the raw request body, grabbing this will be trivial:

```elixir
sig_basestring = "v0:#{timestamp}:#{conn.assigns[:raw_body]}"
```

### 4. Hash the string with the signing secret into a hex signature

Now you need a signature to compare against the one Slack sends you along with the timestamp. Erlang provides a nice crypto library for computing [HMAC-SHA256 keyed hashes](https://en.wikipedia.org/wiki/HMAC), you'll just need to turn that into a hex digest (using `Base.encode16()`):

```elixir
my_signature =
  "v0=#{
    :crypto.hmac(
      :sha256,
      System.get_env("SLACK_SIGNING_SECRET"),
      sig_basestring
    )
    |> Base.encode16()
  }"
```

### 5. Compare the resulting signature to the header on the request

The light is at the end of the tunnel! There are a few more gotchas that aren't exactly intuitive:

1. **`get_req_header` returns an array, even though the same sounds like it returns the singular value**
2. **Leverage the `Plug.Crypto` library to do secure signature comparisons**

As you may have seen from earlier code, `get_req_header` takes in your `conn` and the string of the request header key to return the value...as an array. Not sure why but it's easy to remedy with pattern matching.

Finally, to achieve the `hmac.compare` pseudocode from the Slack tutorial, Elixir has an equal `Plug.Crypto.secure_compare`:

```elixir
[slack_signature] = conn |> get_req_header("x-slack-signature")

Plug.Crypto.secure_compare(my_signature, slack_signature)
```

## Putting it all together

Now that we've solved all the tutorial discrepancies, it's time to put it all together in the context of a Phoenix application.

We've already added the custom raw request body header library and modified our endpoint to accept this `Plug.Parser`. Now we just need to bring the verification into our API controller endpoint you specify within your Slack app dashboard:

```elixir
defmodule MYAPPWeb.SlackController do
  use MYAPPWeb, :controller

  @doc """
  Handle when a user clicks an interactive button in the Slack app
  """
  def actions(conn, params) do
    %{
      "actions" => actions,
      "team" => %{"id" => team_id},
      "type" => "interactive_message",
      "user" => %{"id" => user_id}
    } = Poison.decode!(params["payload"])

    if verified(conn) do
      # do your thing!
      conn |> send_resp(:ok, "")
    else
      conn |> send_resp(:unauthorized, "")
    end
  end

  defp verified(conn) do
    timestamp =
      conn
      |> get_req_header("x-slack-request-timestamp")
      |> Enum.at(0)
      |> String.to_integer()

    local_timestamp =
      :calendar.local_time()
      |> :calendar.datetime_to_gregorian_seconds()

    if abs(local_timestamp - 62_167_219_200 - timestamp) > 60 * 5 do
      false
    else
      my_signature =
        "v0=#{
          :crypto.hmac(
            :sha256,
            System.get_env("SLACK_SIGNING_SECRET"),
            "v0:#{timestamp}:#{conn.assigns[:raw_body]}"
          )
          |> Base.encode16()
        }"
        |> String.downcase()

      [slack_signature] = conn |> get_req_header("x-slack-signature")

      Plug.Crypto.secure_compare(my_signature, slack_signature)
    end
  end
end
```

Congratulations! You can now securely talk with Slack by verifying it's header signature against the one you've generated by hashing your signing secret with the current timestamp. The Slack API documentation is thorough and helpful, but translating it to work in Elixir/Phoenix is not as intuitive as you might imagine.
]]></content:encoded>
      <dc:date>2018-09-30T09:10:00+00:00</dc:date>
    </item>
    <item>
      <title>The algorithms behind the Sort function in JavaScript</title>
      <link>https://www.adamconrad.dev/blog/algorithm-behind-js-array-sort/</link>
      <description><![CDATA[Have you ever wondered what happens when you call Array.prototype.sort()? What did the creators of JavaScript use to sort items in the language? Today we’re going to answer that with our continuing series on common sorting algorithms. But first, the answers to last article’s Daily Problem.

]]></description>
      <pubDate>Tue, 25 Sep 2018 10:28:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/algorithm-behind-js-array-sort/</guid>
      <content:encoded><![CDATA[Have you ever wondered what happens when you call `Array.prototype.sort()`? What did the creators of JavaScript use to sort items in the language? Today we're going to answer that with our continuing series on common sorting algorithms. But first, the answers to last article's Daily Problem.

> This article is part of the [Data Structures and Algorithms Series](/tag/algorithms/) following the course [The Analysis of Algorithms](https://www3.cs.stonybrook.edu/~skiena/373/). If you missed the [previous article](/blog/heapsort-priority-queues-in-js/), check that out first as well as [a copy of _The Algorithm Design Manual_](https://www.amazon.com/Algorithm-Design-Manual-Steven-Skiena/dp/1849967202).

## Daily Problem

> You are given a collection of `n` bolts of different widths, and `n` corresponding nuts.  You can test whether a given nut and bolt together, from which you learn whether the nut is too large, too small, or an exact match for the bolt. The differences in size between pairs of nuts or bolts can be too small to see by eye, so you cannot rely on comparing the sizes of two nuts or two bolts directly. You are to match each bolt to each nut.

Basically this question is asking "does every bolt have a matching nut?"

> 1. Give an `O(n^2)` algorithm to solve the above problem.

The obvious play here seems to be to test every bolt with every nut, which would look like this:

```js
function match(nuts, bolts) {
  let pairs = [];
  for (nut in nuts) {
    for (bolt in bolts) {
      if (nut.fits(bolt) === "exact match") pairs.push([nut, bolt]);
    }
  }
  return pairs;
}
```

There are `n` nuts in a `for` loop checking every `n` bolts in a nested `for` loop, so we can easily determine from our previous discussion on calculating Big O quickly that this algorithm will run in `O(n^2)` time.

> 2. Suppose that instead of matching all of the nuts and bolts, you wish to find the smallest bolt and its corresponding nut. Show that this can be done in only `2n − 2` comparisons.

We start out with `n` nuts and `n` bolts. Start comparing two at random. Likely they will not be the same size, and we will get feedback on that. If something is `too small` for the other, then we keep that item and toss the other piece. So if the nut was `too small` for the bolt, throw away the bolt, grab another one, and then do another comparison. Essentially you will be ping-ponging between nuts and bolts, throwing away all but the last nut and bolt. Since there are `2n` total items, and we throw away all but the last nut (`1`) and the last bolt (`1`), which is `2n-2` comparisons.

What do we do if there is the same? Keep one of either the nut or the bolt aside and do another comparison. If the new nut/bolt is smaller than both, we have our new minimum item and we can throw away both. Regardless, we will get a signal from the new item that will eliminate the kept item or the new item drawn.

> 3. Match the nuts and bolts in expected `O(n log n)` time.

This was one of those Daily Problems that would benefit from reading ahead a bit. In fact, the answer to these 2 parts will become very clear once we get through a few of the algorithms from this article.

## Mergesort: The divide-and-conquer algorithm behind Firefox array sorting

As I mentioned earlier, there is an algorithm that is so good it is used to power the web browser you may be viewing this blog post on. This algorithm is called _mergesort_, and it [was the preferred algorithm](https://bugzilla.mozilla.org/show_bug.cgi?id=224128) for array sorting in Firefox's JavaScript engine over heapsort.

Mergesort works by taking advantage of an essential problem solving technique called **divide-and-conquer**. Practically speaking, divide-and-conquer involves **breaking a problem up into a smaller problem to make it solvable.**

The way we implement divide-and-conquer with mergesort is to _use the power of recursion to halve our problem space_ until all that's left is a simple two item comparison. Once we compare and sort the subarray with two items, we build those lists back up, _merging_ these subarrays together until we have unified the whole array again, sorting our items as we build the list back up.

```js
function merge(leftArr, rightArr) {
  let mergedArr = [];

  while (leftArr.length || rightArr.length) {
    if (leftArr[0] <= rightArr[0]) {
      mergedArr.push(leftArr.shift());
    } else {
      mergedArr.push(rightArr.shift());
    }
  }

  while (leftArr.length) leftArr.shift();
  while (rightArr.length) rightArr.shift();

  return mergedArr;
}

function mergesort(arr) {
  return merge(
    mergesort(arr.slice(0, arr.length / 2)),
    mergesort(arr.slice(arr.length / 2, arr.length))
  );
}
```

### Why use mergesort over heapsort

Truth be told, **mergesort is a very good general sorting algorithm**. Like heapsort, mergesort runs in `O(n log n)` time. If you have the space to copy the array into the sorted array (which you usually do if you aren't in constrained environments like graphics programming or IoT devices), mergesort is great. So if it has the same worst-case runtime as heapsort, why would you choose it over heapsort?

**Mergesort is also excellent for linked lists.** The `merge()` operation of mergesort can be implemented without requiring extra space.

**Mergesort is stable.** The only swapping of items happens when you actually sort within mergesort. In heapsort, since we're using a priority queue, the structure itself is unstable because it's frequently being rearranged to ensure the maximum/minimum value remains at the root of the heap.

## Quicksort: The randomization algorithm powering Chrome's V8 and IE's Chakra

It turns out there are other excellent algorithms for sorting. One such algorithm uses the power of randomization to arbitrarily find a pivot point to sort against. We can use that pivot point recursively just like we do with mergesort to continuously sort smaller and smaller problems until the entire list of items has been sorted. This algorithm is called **quicksort**, and it's used by both [Google](https://github.com/v8/v8/blob/master/src/js/array.js#L328) and [Microsoft](https://github.com/Microsoft/ChakraCore/blob/master/lib/Runtime/Library/JavascriptArray.cpp#L6701) for sorting.

Quicksort and mergesort have quite a bit in common. They both build a nested recursion tree all the way down to every `n` items within a list, each of which takes linear time to process. That means they both generally run in `O(nh)`, where `h` is equal to the height of the recursion tree that is generated.

Where they differ lies in the randomization of the pivot point. Whereas with mergesort we always subdivide our arrays evenly, in quicksort that point is never known until the function is run.

In the best case, we pivot on the median point, halving the array each time leading us to a runtime of `O(log n)` and is on par with heapsort and mergesort. In the worst case, we pivot on the end of the array with a completely unbalanced tree that only ever sorts 1 item at a time giving us runtime of `O(n^2)` and is on par with selection sort.

So if the worst case is so bad, why would we bother using quicksort? The answer lies in the average case random sampling.

With mergesort the best and only case is the exact median of the list. If the list isn't perfectly _unsorted_, you could be suboptimally generating your recursion sorting tree.

Similarly, with premeditated data, we know that we can find a configuration that gets us the best- and worst-case scenarios. But since unsorted data can be in any random order, how does quicksort help us?

With quicksort, the middle _half_ of the list (e.g. for `[1,2,3,4]`, that's `[2,3]`) is a good enough pivot to not hit the worst-case runtime. The reason is a pivot somewhere in the middle still generates a recursive tree that is some logarithmic order of magnitude (specifically `2n ln n`), the kind of speed we're looking for.

To solidify a `O(n log n)` runtime, we grab a _random_ permutation of our initial list which takes `n` time. And since _random_ data can be sorted on average in `O(log n)`, adding in this randomization step _guarantees_ we can provide an `Θ(n log n)` runtime with _any_ input.

```js
function partition(arr, pivotIndex, lower, higher) {
  let pivotPoint = arr[pivotIndex];
  let partitionIndex = lower;

  for (let i = lower; i < higher; i++) {
    if (arr[i] < pivotPoint) {
      [arr[i], arr[partitionIndex]] = [arr[partitionIndex], arr[i]];
      partitionIndex++;
    }
  }

  [arr[right], arr[partitionIndex]] = [arr[partitionIndex], arr[right]];

  return partitionIndex;
}

function quicksort(arr, lower, higher) {
  let pivotIndex = higher;
  let partitionIndex = partition(arr, pivotIndex, lower, higher);

  quicksort(arr, lower, partitionIndex - 1);
  quicksort(arr, partitionIndex + 1, higher);
}
```

### Why use quicksort

Now we have three algorithms with an expected runtime of `O(n log n)`. Why would we choose quicksort?

**Quicksort performs in-place sorting.** Like heapsort, quicksort requires no extra space.

Mergesort requires an additional array to copy your merged items into. If you have space constraints, quicksort beats mergesort.

**Quicksort beats mergesort for arrays.**

Since quicksort takes advantage of random access, we already know that the constant-time access of arrays by index makes it very easy to use a random number and retrieve an item quickly. With linked lists, the random access no longer pays off, since all items must be accessed from either the front or the back of the list.

**Quicksort is great for sorting arrays with hundreds of thousands of unique keys**.

In fact, when benchmarked against 100,000 unique keys or more, [quicksort outperformed mergesort and heapsort by 1.5-2x](http://alejandroerickson.com/j/2016/08/02/benchmarking-int-sorting-algorithms.html).

### Revisiting the Daily Problem

With the available algorithms at our disposal, this should make the Daily Problem feasible to answer beyond the naive first section.

> Match the nuts and bolts in expected `O(n log n)` time.

With our randomized quicksort algorithnm in place, we can now sort both nuts and bolts by size, and both sets will hvae a pairing because the first element of one matches the first element of the other, and so on all the way up to the `n`th element. If this still doesn't make sense, check out an [in-depth analysis of randomized quicksort](https://www.cs.cmu.edu/~avrim/451f11/lectures/lect0906.pdf).

## Bucketsort: Sorting via distribution

With three sorting algorithms in our optimal band of `O(n log n)` already, how could we possibly get any better? Consider the following example:

Rather than a list of numbers, you now have a list of names, like in your iPhone. How do you find a name in your phone? If you scroll really quickly, you'll see a letter highlighting roughly which letter of the alphabet you're in. From there you'll pretty much go through the whole alphabet starting with the second letter of the name until you find the person you're looking for. _This is exactly how bucketsort works (also known as distribution sort or binsort).

**Bucketsort is great for evenly-distributed chunks of similar data.** If we used bucketsort in the above example, we would first partition the whole phone book into 26 subarrays (well maybe a few less because letters like _X_ and _Z_ have far fewer names than _M_ or _S_).

I hope you're seeing a theme here. We once again are _subdividing the problem_; instead of dividing equally in halves or by a random pivot, we branch a list of `n` items by `k` logical partitions based on how the data is arrange. Now obviously we can't implement a generic algorithm for bucketsort in JavaScript because it highly depends on the dataset. Bucketsort for last names will look very different from bucketsort on tabular accounting data for a business.

## Binary search: The algorithm that powers BSTs

Now that we have a gaggle of sorting algorithms to choose from, what can we do with this sorted data to ensure we're still operating efficiently? **Binary search is the `log n` application for searching sorted data.**

All we have to do is start with the middle element. If it's not the middle element, we ask "is our target item less than or greater than the middle element?" We then choose a side and repeat the process until our target element is found:

```js
function binarySearch(arr, target) {
  let middle = Math.floor(arr.length / 2);

  if (arr[middle] === target) return middle;
  if (arr[middle] < target) {
    binarySearch(arr.slice(middle + 1), target);
  } else {
    binarySearch(arr.slice(0, middle - 1), target);
  }
}
```

### Meta binary search

What happens if you have an unbounded sorted array? For example, we can create an unbounded array using [ES6 generators](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/function*). The fibonacci sequence is unbounded, and can be lazily evaluated step-by-step using generators. How would we find an item within such an array?

**Meta binary search**, or one-sided binary search, to the rescue! This search is twice is slow because it requires two `log n` searches. The first search is find our element within some bounds. So we start with a single-element array. Then a two-element array. Then a four-element array. And so on and so on, doubling our bounded subarray until we find our element within some boundary. Once we find the element, we can then proceed with our normal binary search.

### Bisection method / numerical analysis

Do you have to divide things in half? The **bisection method** (which you can read more about [here](https://en.wikipedia.org/wiki/Bisection_method)) is a way finding the roots of values by repeatedly _bisecting_ an interval until the target is found. In other words, this is a more generalized form of binary search and can operate on the square root, cubed root, or any root of a number.

[Numerical analysis](https://en.wikipedia.org/wiki/Numerical_analysis#Computing_values_of_functions) is deeply rooted in mathematics. **If you're doing graphics programming or machine learning, you may find yourself utilizing the bisection method.**

## Bonus sorts: Shellsort & Radix Sort

I wanted to cover two other sorting algorithms really quick because they're super interesting and are great advanced algorithms for specific situations. They aren't really covered in the _Algorithm Design Manual_ but they do have practical applications.

**Shellsort** is like an optimized version of either _bubble sort_ (where you continuously swap smaller elements with larger elements) _insertion sort_ (where you insert unsorted elements into the sorted portion of an array), depending on how you look at it. Shellsort sorts in intervals known as _gaps_, and iteratively reduces the number of gaps it sorts by until the whole array is sorted.

For example, for an array of a randomized ordering of natural numbers (integers greater than 0) up to 100, shellsort would first sort the numbers [5,10,15,20,...100]. Then it would sort [3,6,9...99]. And that _gap_ interval would reduce down until it got to a gap of 1. But by the time it was sorting against every integer, all of the 5s were sorted, and all of the 3s were sorted, leaving only a fraction of the actual numbers left to sort.

Like bucketsort, shellsort requires a custom gap sequence depending on the application of your sorting algorithm. You can also choose either insertion sort or bubble sort as the inner function to perform on your gap subset. You can find an example [here](https://en.wikipedia.org/wiki/Shellsort#Pseudocode), but like I said, it's difficult to really nail in the general sense since it is highly dependent on the data you are sorting.

**Radix sort** (of which there are actually two) is arguably one of the fastest sorting algorithms known. If you really dove into the benchmarking link I posted above, it outperforms every other sorting algorithm we've mentioned here so far. Why is that the case?

At the end of the day everything is binary. Integers, strings, floating-point numbers, and everything else on a computer is all represent as binary numbers. Because of that, sorting objects by their binary code is a really convenient way to sort objects. Because of that, radix sort is really two algorithms, where you can sort starting on the least-significant digit or the most-significant digit of our object, now represented as a number.

One great example of using radix sort (specifically the MSD version) would be to [find the occurrence of a word](https://en.wikipedia.org/wiki/Radix_sort#trie-based_radix_sort) using tries to construct a word dictionary and then depth-first search to find the word you are looking for, starting with the first (or most significant) letter.

There are lots of other sorting algorithms that you can explore. As we move on from simple data structures and sorting to graphs and advanced tree searching, we'll demystify what exactly are things like tries and depth-first search. But this section was exhausting, so let's end with the Daily Problem and call it a day for now.

## Even more practice

If you just can't get enough of sorting problems, here are some more problems from the book to flex your sorting muscles:

1. 4-20
2. 4-22
3. 4-24
]]></content:encoded>
      <dc:date>2018-09-25T10:28:00+00:00</dc:date>
    </item>
    <item>
      <title>Why Airbnb has pivoted away from React Native</title>
      <link>https://www.adamconrad.dev/blog/why-airbnb-has-pivoted-away-from-react-native/</link>
      <description><![CDATA[For the last two years, Airbnb has been using React Native to accelerate development of their mobile platforms. React Native is an open source framework developed by Facebook that allows JavaScript developers to create familiar React components that work across web, Android, and iOS development environments.

]]></description>
      <pubDate>Thu, 20 Sep 2018 10:28:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/why-airbnb-has-pivoted-away-from-react-native/</guid>
      <content:encoded><![CDATA[For the last two years, Airbnb has been using React Native to accelerate development of their mobile platforms. React Native is an open source framework developed by Facebook that allows JavaScript developers to create familiar React components that work across web, Android, and iOS development environments.

Recently, Gabriel Peal [wrote an article](https://medium.com/airbnb-engineering/react-native-at-airbnb-f95aa460be1c) outlining Airbnb’s decision to move off of React Native by 2019 in favor of native mobile development tools. We sat down with Gabriel to discuss the history of mobile development at Airbnb, the introduction of React Native, its benefits and drawbacks, and ultimately why Airbnb decided to move on from React Native.

## A timeline of mobile development at Airbnb

![](https://cdn-images-1.medium.com/max/2000/0*QTNKnQLRTSBoaOv6)

Originally Airbnb was just a website with increasing traffic on mobile platforms. By 2012, the engineering team recognized that in order to support the demands of its mobile user base, they would need to invest additional resources into mobile platforms. For four years the team built out iOS and Android applications with dedicated mobile developer resources and infrastructure.

However, by 2016, the landscape had shifted so far toward mobile that the development team could not keep up with demand. Without the necessary talent for mobile developers, and a vast majority of developers trained on the web, a few teams began to look at cross-platform solutions.

Airbnb’s web front-end is built with React, and the vast majority of developers at Airbnb are comfortable with this popular web framework. In 2016 Facebook launched React Native after using it internally with their iOS and Android apps. A few developers on the team saw this as a great way to leverage their existing web development team to assist in development of their mobile applications. A few months later, with a test pilot app ready, Airbnb launched their first set of components using React Native to support their iOS and Android native applications.

React Native rapidly increased the development cycle of their mobile applications, and further investments were made into developing, supporting, and providing infrastructure to its 800,000-line mobile codebase.

## Benefits and drawbacks of React Native

![](https://cdn-images-1.medium.com/max/2000/0*pjcJTIj2UKtKvZmz)

Clearly, the biggest benefits to React Native for Airbnb were the [rapid development and platform agnosticism](https://softwareengineeringdaily.com/2017/04/11/the-future-of-react-native-with-brent-vatne-and-adam-perry/). With React Native, Airbnb could leverage its entire development team of engineers to help build and support their mobile applications. This also meant that future hires could also be trained regardless of their background in mobile or web. It also meant a leaner codebase, because one repository could be deployed across three platforms. But with these benefits come challenges as well. Without being prescriptive in his advice to use (or not use) React Native, Peal offers these considerations when deciding to go with React Native:

## 0 to 1 is a major investment

A common misconception is that when Airbnb decided to use React Native, that they made a complete switch. This is not only not true, but to this day the React Native code represents only 15–20% of the codebase and developer resources. It was never a majority platform for the company, and part of that was the immense upfront cost to enable even the first set of components for the mobile platforms.

Even though React Native is a framework and vastly simplifies mobile development, it is not trivial to get up and running, particularly if you have an existing codebase. **Expect to invest a substantial amount of time and energy to integrating React Native into your existing mobile codebase.**

## Developers will still need to write native code

React Native is a great choice for simple applications where the APIs have a clear bridge between two platforms. In the earlier days of the Airbnb mobile apps, the platform was a great accelerator for the simpler tools within the mobile application. Eventually, the APIs will not operate exactly the way you want and you will have to dive into the native libraries to make the necessary adjustments.

Some examples where native bridges will need to be written include the networking stack, experimentation, internationalization, deep links, navigation, and advance map features.

If you hire a JavaScript developer to work on your React Native project, **expect that they will have to write native code to bridge any gaps in functionality.**

## Beware of landmines

Peal warns of “landmines, or interactions in the code that can be extremely difficult to track down and reproduce. One example involved React Native components rendering all white on certain phones. This bug was not only unreliable but difficult to track down.

The solution was to turn off initializing [Fresco](http://frescolib.org/), the standard React Native image rendering library for Android. To this day, the engineers still do not understand why this fixed the bug. **If Airbnb can run into landmines that turn seemingly simple tasks into weeks-long bug hunts, expect you will, too.**

## Investment is never-ending and slow

Many teams will prototype a small mobile application with React Native and immediately believe that it will solve all of their problems. But like all open source software, the platform is ever-changing with new bugs and issues discovered daily. **Expect to make continuous investments to support the volatile landscape of the React Native ecosystem.**

One example included a patch that Airbnb engineers wanted to get into the React Native framework as soon as possible to align with their product goals. Unfortunately, React Native’s development lifecycle only pushes out a release once every four weeks. If they timed their fix wrong, they would have to wait almost a month before their changes could be adopted by the React Native framework. With the speed of development at Airbnb, they simply could not afford to take that risk.

## Alternatives to React Native

![](https://cdn-images-1.medium.com/max/2000/0*4fuJ9LhmxaGslsK6)

Ultimately the drawbacks outweighed the benefits, which prompted Airbnb to reconsider, and ultimately withdraw from React Native development. While support will continue through the end of the year, Airbnb plans to discontinue active development of React Native projects by early 2019.

Even if you still are not sold on the value of native-first development for mobile applications, there are other cross-platform alternatives besides React Native. [Flutter](https://flutter.io/) is Google’s answer to React Native and has been previously covered on Software Engineering Daily (you can check out more from previous shows [here](https://softwareengineeringdaily.com/2018/07/09/flutter-with-eric-seidel/) and [here](https://softwareengineeringdaily.com/2018/07/11/flutter-in-practice-with-randal-schwartz/)). [Xamarin](https://visualstudio.microsoft.com/xamarin/) is Microsoft’s version of React Native, and additionally provides support for Windows mobile phones as well.

While both frameworks are cross-platform, similar issues can be found like the ones outlined above. Additionally, both frameworks have relatively low adoption within Google and Microsoft, respectively. Ultimately, while cross-platform frameworks solve the majority of simple problems, anything more complicated will ultimately require you to resort back to the native platforms of a given mobile device.

*Originally published at [softwareengineeringdaily.com](https://softwareengineeringdaily.com/2018/09/24/show-summary-react-native-at-airbnb/) on September 24, 2018.*
]]></content:encoded>
      <dc:date>2018-09-20T10:28:00+00:00</dc:date>
    </item>
    <item>
      <title>Heapsort using Priority Queues in JavaScript</title>
      <link>https://www.adamconrad.dev/blog/heapsort-priority-queues-in-js/</link>
      <description><![CDATA[As we discussed in the last article, sorting is the most frequent activity a computer performs, and also happens to be the most thoroughly-studied activity by computer scientists. Over the next few articles, we’re going to explore the tried-and-true sorting algorithms have that defined the field throughout the years, introducing relevant advanced data structures along the way.

]]></description>
      <pubDate>Thu, 20 Sep 2018 10:28:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/heapsort-priority-queues-in-js/</guid>
      <content:encoded><![CDATA[As we discussed in the last article, sorting is the most frequent activity a computer performs, and also happens to be the most thoroughly-studied activity by computer scientists. Over the next few articles, we're going to explore the tried-and-true sorting algorithms have that defined the field throughout the years, introducing relevant advanced data structures along the way.

> This article is part of the [Data Structures and Algorithms Series](/tag/algorithms/) following the course [The Analysis of Algorithms](https://www3.cs.stonybrook.edu/~skiena/373/). If you missed the [previous article](/blog/intro-to-sorting/), check that out first as well as [a copy of _The Algorithm Design Manual_](https://www.amazon.com/Algorithm-Design-Manual-Steven-Skiena/dp/1849967202).

## Daily Problem Explained

> Give an efficient algorithm to determine whether two sets (of size `m` and `n`) are disjoint. Analyze the complexity of your algorithm in terms of `m` and `n`. Be sure to consider the case where `m` is substantially smaller than `n`.

This is just another exercise in writing pseudocode; a great problem for a programming interview! The only real catch here is you have to know what a [disjoint set](https://en.wikipedia.org/wiki/Disjoint_sets) is. A **disjoint set** is really just _two collections that have no items in common_. An example for this problem would be _`m = [1,2,3], n = [4,5,6]`_.

Let's just start throwing stuff out there and see what sticks. My first thought is that when you  combine set of two disjoint sets is of length `m + n`, since no duplicates exist. That's pretty important because if your final set `S` is such that `S.length < m.length + n.length` you fail the disjoint part of the problem. Can we exploit that? Here's what I'm thinking:

```js
function isDisjoint(m, n) {
  let finalSet = m;
  for (let item in n) {
    if (!finalSet.includes(item)) finalSet.push(item);
  }
  return finalSet.length === (m.length + n.length);
}
```

On first glance you only see 1 `for` loop so it looks like it's `O(n)` which is fast! But then we quickly realize the absolute fastest we can go is `O(n+m)` since we have to evaluate all items to duplication, so something is up. T

The key here is that `includes` has to inspect the entire `m` list for each `n` items. That leaves us with an algorithm of `O(mn)`. It's not bad but it's not great either; remember, that's roughly equivalent to `O(n^2)` (assuming `m` and `n` were the same length) which is quadratic. Quadratic functions are fine for relatively small data sets, but as we noted [in our article on Big O Notation](/blog/big-o-notation/), once we go above a million items in our dataset, the algorithm will be too slow. Can we improve on this?

One thing I quickly realized is we can leverage what we've learned so far as a hint on how to solve this. What data structures have we been studying carefully? Which of those are praised for their performance?

Then it hit me: **we can use a hash table to keep track of the number of times we've touched an item and if any already exist we know right away the set is not disjoint and we can exit early**.

Adding to and searching from a hash table takes an expected `O(1)` time, `O(n)` at worst. That means we can hit that magical runtime of `O(m+n)` in the worst-case using a hash table!

```js
function isDisjoint(m, n) {
  let finalSet = new HashTable();
  for (let item in m) {
    finalSet.insert(item);
  }
  for (let item in n) {
    if (finalSet.search(item)) {
      return false;
    } else {
      finalSet.insert(item);
    }
  }
  return true;
}
```

## Standard Selection Sort in JavaScript

The first sorting algorithm we're going to look at is **selection sort**. Selection sort's algorithm is very simple: _continue to take the smallest item from one array a place it into a new array_. By virtue of taking the smallest item from a recursively-shrinking array, you're left with a new array that is sorted. A naive implementation using standard JavaScript library functions looks like this:

```js
function selectionSort(arr) {
  let sorted = [], min;
  for (let i = 0, n = arr.length; i < n; i++) {
    min = Math.min(...arr);
    sorted[i] = min;
    arr.pop(min);
  }
  return sorted;
}
```

As we can see from the algorithm above, selection sort performs `n` iterations, but each iteration will take roughly `n/2` steps in order to find the minimum value (we know this because finding a minimum value takes at worst `O(n)` but our problem space is being reduced with each successive iteration), giving us a grand total **worst-case runtime of `O(n^2)`**. Is there a way we can improve on this runtime?

## Selection Sort + Priority Queues = Heapsort

Look back to our previous algorithm - what are the primary operations for selection sort? Each iteration consists of a `min()`, an `insert()`, and a `delete()`. There is, in fact, a special data structure that combines `delete()` with `min()` to _efficiently_ extract the minimum value from its collection. This data structure is called a **priority queue**.

### Priority queues

Priority queues are flexible data structures that _allow for elements to enter the queue at arbitrary intervals while exiting in a sorted fashion_. Priority queues support three primary operations: _insert_, _findMin_ (or _findMax_), and _extractMin_ (or _extractMax_). For the purposes of simplicity going forward, we're going to focus on priority queues that implement the _minimum_ rather than the _maximum_ value.

A summary of their worst-case runtimes can be found below. **As an exercise: can you explain how these values are calculated?** Here's a hint: `findMin()` is always constant-time complexity because you can always use a variable to keep track of the minimum value as new values are inserted.

<table>
  <thead>
    <th>Data structure / method</th>
    <th><code>findMin()</code></th>
    <th><code>extractMin()</code></th>
    <th><code>insert()</code></th>
  </thead>
  <tbody>
    <tr>
      <td><strong>Unsorted array</strong></td>
      <td><code>O(1)</code></td>
      <td><code>O(n)</code></td>
      <td><code>O(1)</code></td>
    </tr>
    <tr>
      <td><strong>Sorted array</strong></td>
      <td><code>O(1)</code></td>
      <td><code>O(1)</code></td>
      <td><code>O(n)</code></td>
    </tr>
    <tr>
      <td><strong>BST</strong></td>
      <td><code>O(1)</code></td>
      <td><code>O(log n)</code></td>
      <td><code>O(log n)</code></td>
    </tr>
  </tbody>
</table>

### Implementing `extractMin()` and `insert()` with heaps

As I mentioned earlier, there is a special data structure that can efficiently handle the primary operations of selection sort. This data structure is known as a **heap**. Heaps are a variant of a priority queues that keep a rough track of the order of its items. This **ordering is smart enough to keep track of the minimum/maximum value, but not a complete sort which would significantly slow down the data structure**.

To implement a heap, we are effectively creating a _binary tree without worrying about pointers_, or in other words _an array of keys where the index of the array item _is_ the pointer_. The root of the tree is the first item in the array (and also the minimum value), with its left and right children corresponding to the 2nd and 3rd values in the array. In general, it looks something like this:

```js
class PriorityQueue {
  const PRIORITY_QUEUE_SIZE = 500; //arbitrarily picked

  constructor(arr, n) {
    this.queue = new Array(PRIORITY_QUEUE_SIZE);
    this.length = 0;

    for (let i = 0; i < n; i++) {
      this.insert(arr[i]);
    }
  }

  parent() {
    if (this.length === 0) return -1;
    return Math.floor(this.length / 2);
  }

  child(n) {
    return 2 * n;
  }
}
```

Remember: we don't care about what's inside of each slot in our priority queue, we just care about finding elements at specific locations. That is why we can calculate locations with just some multiplication and division rather than having to search by value. Because we're not pointing to any specific values, the index of our elements is a reflection of the math we are performing, and when we perform basic operations like `*` and `/`, we know we're dealing with fast computational operations in constant time.

The downside to a heap is that we can't really do an efficient search in a heap. Because we're only concerned about keeping the smallest item at the root, everything else is arbitrarily packed into the tree. The tree items are packed to maintain space efficiency, but remember this isn't a binary _search_ tree, just a binary tree (i.e. each node has two children).

#### Insertion

As we mentioned earlier, save for the smallest item, the heap is just packed in from the top down, filling in from left to right before moving down to the next layer of the tree. This keeps the heap efficiently packed. To ensure the minimum-dominance relationship is maintained, we have to make sure we can swap up the tree as far as necessary to satisfy our minimum constraints.

```js
class PriorityQueue {
  // ...

  insert(item) {
    if (this.length >= PRIORITY_QUEUE_SIZE) {
      throw new Error("Priority queue overflow");
    } else {
      this.length += 1;
      this.queue[this.length] = item;
      this.bubbleUp(this.length);
    }
  }

  bubbleUp(parent) {
    if (this.parent(parent) === -1) return;

    let currentParrent = this.queue[this.parent()];
    let parentContender = this.queue[parent];

    if (currentParrent > parentContender) {
      [currentParent, parentContender] = [parentContender, currentParrent];
      this.bubbleUp(parent);
    }
  }
}
```

Analyzing the above two functions, we see that `insert()` has no loops and emulates a tightly-packed binary tree. We know the height of a binary tree is `lg n` (shorthand for `log n` of base 2), so our worst-case runtime is `O(log n)`.

The `bubbleUp()` has the primary function of swapping elements (remember from ES6 the swap appears as `[x,y] = [y,x]`) which takes constant time. Therefore, to create a heap with `n` items it will take `O(n log n)` to insert and construct.

#### Extracting the minimum value

Finding the minimum value is easy, since it is the root of the tree. Actually pulling it out of the array is the hard part, since this leaves the tree broken in two at the root. To handle this, we'll need to re-assign the right child of the root's children to be the new root.

Why the right-most element? Because _we pack from left to right, so we want to unfurl our array in a clean manner from the opposite direction_. If the value isn't correct in the minimum-dominance relationship, we can always bubble _down_ the array (or "heapify") to swap out for the correct minimum.

```js
class PriorityQueue {
  // ...

  extractMin() {
    let min = -1;

    if (this.length <= 0) throw new Error("No minimum: Empty priority queue");

    min = this.queue[0];
    this.queue[0] = this.queue[this.length];
    this.length -= 1;
    bubbleDown(1);

    return min;
  }

  bubbleDown(parent) {
    let childIndex = child(parent), minChild = parent;

    for (let i = 0; i <= 1; i++) {
      if (childIndex <= this.length) {
        if (this.queue[minChild] > this.queue[childIndex + i]) {
          minChild = childIndex + i;
        }
      }
    }

    if (minChild !== parent) {
      [minChild, parent] = [parent, minChild];
      bubbleDown(minChild);
    }
  }
}
```

Similarly for extracting the minimum value, bubbling down only goes down a tree height of `lg n`, so at worst finding and removing the minimum value will take `O(log n)` time.

## Selection Sort using Priority Queues as Heaps

Putting it all together, we now have our optimized selection sort, known as _heapsort_, which combines everything we've learned here into an efficient extraction machine:

```js
function heapsort(arr, n) {
  let heap = new Heap(arr, n);

  for (let i = 0; i < n; i++) {
    arr[i] = heap.extractMin();
  }

  return arr;
}
```

How elegant! And the best part: since our priority queue's operations run in `O(log n)` time, over a set of `n` elements, **heapsort improves on selection sort from `O(n^2)` to `O(n log n)`**, the gold standard for sorting algorithm time efficiency!

Even better, we did it _in-place_, meaning we didn't have to instantiate a second array to copy the elements over. We were able to use the same array to swap elements and create our sorted list of items.

Overall, this meaps heapsort an excellent algorithm for both its space and time efficiency.

Now there is a _slightly_ more advanced way of designing the constructor such that we populate the first `n` positions in heap directly from our input array (meaning we've created a one-sided heap) and then use a bunch of `bubbleDown()` calls (`n/2` to be exact since half of the items are already on the correct side of their parents) to balance out the array and find our minimum. This speeds up construction from `O(n log n)` to `O(n)` on average, but since construction is not the main operation of a heap, and the worst-case operations still take `O(n log n)`, this isn't too big of a win, **but you're welcome to implement it for yourself if you're looking for an extra exercise in algorithm design**.

## Wrapping up with the next Daily Problem

We've covered a lot with selection sort, priority queues, and heapsort. We've seen we can pretty easily cover sorting in `O(n^2)` time, but with a few optimizations, we can improve that time to the optimal `O(n log n)`. This will be useful in tackling today's Daily Problem:

> You are given a collection of `n` bolts of different widths, and `n` corresponding nuts.  You can test whether a given nut and bolt together, from which you learn whether the nut is too large, too small, or an exact match for the bolt. The differences in size between pairs of nuts or bolts can be too small to see by eye, so you cannot rely on comparing the sizes of two nuts or two bolts directly. You are to match each bolt to each nut.
>
> 1. Give an `O(n^2)` algorithm to solve the above problem.
>
> 2. Suppose that instead of matching all of the nuts and bolts, you wish to find the smallest bolt and its corresponding nut. Show that this can be done in only `2n − 2` comparisons.
>
> 3. Match the nuts and bolts in expected `O(n log n)` time.

Think you have an answer? Create a [gist](https://gist.github.com) and send it to me [on Twitter](https://twitter.com/theadamconrad) and let's compare notes!

## More problems to practice on

Now that [homework 2](http://www3.cs.stonybrook.edu/~skiena/373/hw/hw2.pdf) is out, here are a few problems that are relevant to the sorting we've discussed so far:

1. 4-1
2. 4-2
3. 4-5
4. 4-6
5. 4-12
6. 4-13
7. 4-14
8. 4-15
]]></content:encoded>
      <dc:date>2018-09-20T10:28:00+00:00</dc:date>
    </item>
    <item>
      <title>Introduction to sorting algorithms in JavaScript</title>
      <link>https://www.adamconrad.dev/blog/intro-to-sorting/</link>
      <description><![CDATA[Now that we’ve covered basic data structures, it’s time to apply those structures to one of the most basic applications of algorithms: sorting. Sorting, as the name implies, organizes and divides objects for us in order to make them easier to find and use.

]]></description>
      <pubDate>Tue, 18 Sep 2018 10:28:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/intro-to-sorting/</guid>
      <content:encoded><![CDATA[Now that we've covered basic data structures, it's time to apply those structures to one of the most basic applications of algorithms: sorting. Sorting, as the name implies, organizes and divides objects for us in order to make them easier to find and use.

> This article is part of the [Data Structures and Algorithms Series](/tag/algorithms/) following the course [The Analysis of Algorithms](https://www3.cs.stonybrook.edu/~skiena/373/). If you missed the [previous article](/blog/hashing-in-js/), check that out first as well as [a copy of _The Algorithm Design Manual_](https://www.amazon.com/Algorithm-Design-Manual-Steven-Skiena/dp/1849967202).

Over the next several articles in this series, we are going to implement many of these basic sorting algorithms in JavaScript so you can compare and contrast the approaches and even find which are used in the JavaScript language today.

## Answers to the Daily Problem

This round our Daily Problem is back to coming directly from the book, practice problem 4-3:

> Take a sequence of `2n` real numbers as input. Design an `O(n log n)` algorithm that partitions the numbers into `n` pairs, with the property that the partition minimizes the maximum sum of a pair. For example, say we are given the numbers `(1,3,5,9)`. The possible partitions are `((1,3),(5,9))`, `((1,5),(3,9))`, and `((1,9),(3,5))`. The pair sums for these partitions are `(4,14)`, `(6,12)`, and `(10,8)`. Thus the third partition has 10 as its maximum sum, which is the minimum over the three partitions.

There's a lot of words here but we can reason about this another way: the pattern here is to minimize the largest sum is pairing the maximum number with the minimum number, and working inward.

We first need to sort (which takes `n log n`) the numbers, and then pair them off by taking the `maximum()` and the `minimum()` number from the set, plucking them out of the old set and placing them into a new set. Finally, we take the largest number from those pairs and that's the maximum sum that is the minimum across all possible permutations of the problem. In pseudocode, it looks something like this:

```js
function minimumMaxSum(set) {
  let sortedSet = set.sort();
  let pairs = [];

  while (sortedSet) {
    let maximum = sortedSet.maximum();
    let minimum = sortedSet.minimum();
    pairs.push(maximum + minimum);
    sortedSet.remove(maximum);
    sortedSet.remove(minimum);
  }

  return pairs.maximum();
}
```

## Applications of sorting

If you're wondering why sorting gets so much attention in interviews, it's because sorting is so important and so common. In fact, it is reported that [nearly a quarter of all computational time](https://www.johndcook.com/blog/2011/07/04/sorting/) is spent in just sorting data. Because it's so common, we have developed dozens of algorithms to tackle this problem, and is in fact one of the most studied problems in computer science. Here are just a few of the common use cases for sorting:

* **Searching:** If you want to find stuff, it's going to be way easier to find stuff if it's organized. Can you imagine searching for books in a library if everyone just stuffed their books in there arbitrarily?
* **Frequency / Uniqueness:** Remember things like _mean_, _median_, and _mode_ from math class? It's much easier to find the _mean_ and _median_ if you know they are roughly in the middle of your set. Similarly, the _mode_ will always be grouped together in the largest group of a set; all made easier by sorting.
* **Selection:** Like search, picking something out of a set is much easier when it's in a group of sorted items. Otherwise, the thing you want to pick is in a random group and may take longer to find.

## Approaches to sorting

When you think of sorting items, what comes to mind? Probably something like largest to smallest (or vice versa), and probably involving numbers. But lots of data can be sorted in lots of different ways, so before we dive into sorting algorithms it's important to remember that this is not the only way to sort. Here are a few more examples to consider when using sorting algorithms:

* **By key/record:** Just because we have the largest/smallest piece of data doesn't mean it is special. Sometimes it has to do with when/where that data was entered. When we enter data into a database with a unique incremental ID, we can sort by created-at date using the key of that record.
* **By alphabetical ordering:** Numbers increase and decrease, but so do letters based on their ordering in an alphabet. Even though there is no mathematical basis for A to be "less than" Z, there is a semantic and contextual basis.
* **By frequency:** Even if something isn't the newest or the largest, we may want to have the most popular/frequent data at the top to show to customers. Grouping by frequency and sorting on that can also be very important in picking out outliers.

Just like applications, the approaches to sorting are also numerous. What's important is not that this list is exhaustive, but that it is a _starting point for thinking about sorting as a fundamental activity of computer programming_. Whether or not you're a web programmer or an AI programmer, you _will_ have to sort some data while doing your job. Knowing how that works under the hood is advantageous, particularly if you're needing to optimize your programs for performance.

**One caveat: just because you should learn how this works doesn't mean you should implement your own sorting algorithms in a professional setting.**

This is important because while programming interviews may ask you to implement algorithms to commonly-solved problems, that doesn't mean you should _actually_ use them in the real world. What is important is that you know _how_ it works, not that you need this in your arsenal of tools to implement.

Next time, we'll look at one of these high performers that uses priority queues, a special variant on queues that we'll explore in-depth while we spell out our first sorting algorithm.

## Next Daily Problem

This one is pretty straightforward question, sort of something like you'd see out of a programming interview!

> Give an efficient algorithm to determine whether two sets (of size `m` and `n`) are disjoint. Analyze the complexity of your algorithm in terms of `m` and `n`. Be sure to consider the case where `m` is substantially smaller than `n`.

Think you have one in mind? Throw it up on a [gist](https://github.com/gist) and send it to me [on Twitter](https://twitter.com/theadamconrad)!
]]></content:encoded>
      <dc:date>2018-09-18T10:28:00+00:00</dc:date>
    </item>
    <item>
      <title>Hashing in JavaScript</title>
      <link>https://www.adamconrad.dev/blog/hashing-in-js/</link>
      <description><![CDATA[We know that dictionaries are very important data structures. But did you know they can be improved on from our previous implementations using arrays and lists? With a concept called hashing, we can create an excellent implementation of dictionaries that efficiently handle all of that linked memory.

]]></description>
      <pubDate>Thu, 13 Sep 2018 10:28:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/hashing-in-js/</guid>
      <content:encoded><![CDATA[We know that dictionaries are very important data structures. But did you know they can be improved on from our previous implementations using arrays and lists? With a concept called _hashing_, we can create an excellent implementation of dictionaries that efficiently handle all of that linked memory.

> This article is part of the [Data Structures and Algorithms Series](/tag/algorithms/) following the course [The Analysis of Algorithms](https://www3.cs.stonybrook.edu/~skiena/373/). If you missed the [previous article](/blog/dictionaries-in-js/), check that out first as well as [a copy of _The Algorithm Design Manual_](https://www.amazon.com/Algorithm-Design-Manual-Steven-Skiena/dp/1849967202).

## Answers to the Daily Problem

To solidify our understanding of dictionaries, the Daily Problem focuses on utilizing the primary functions of dictionary objects:

> You are given the task of reading in `n` numbers and then printing them out in sorted order. Suppose you have access to a balanced dictionary data structure, which supports each of the operations search, insert, delete, minimum, maximum, successor, and predecessor in `O(log n)` time.

So basically it's asking us to use some creative ju-jitsu with dictionaries in various situations.

> 1. Explain how you can use this dictionary to sort in `O(n log n)` time using only the following abstract operations: minimum, successor, insert, search.

Sorting in `n log n` is a pretty common worst-case for good sorting algorithms. It basically means we have to touch all `n` numbers and then divide and conquer the sorting part (which is `log n`). With these 4 functions available, we can sort with something like this:

The book does this slightly differently - first it adds all of the numbers to the list, then it sorts it using the `successor()` function, but I'm assuming we already have a list that we can copy into a new one by starting with the `minimum()` object and inserting it, _presorted_, into the new dictionary.

```js
function sort(n) {
  let m = 0, min = minimum(n), newDict = new Dictionary();
  while(m !== n) {
    newDict.insert(min);
    min = n.successor(min);
    m++;
  }
  return newDict;
}
```

> 2. Explain how you can use this dictionary to sort in `O(n log n)` time using only the following abstract operations: minimum, insert, delete, search.

In this case, we keep stripping off the smallest item from the old list of `n` numbers. Each time we grab the minimum, we add it into the new dictionary, and then delete it from the old, unsorted dictionary.

This is actually a perfect example of what the book refers to as _recursive data structures_. When we remove one item from the dictionary, we are left with just a smaller dictionary. We can exploit this by recursively grabbing the smallest item and adding it to our new dictionary to create a sorted object.

```js
function sort(n) {
  let min, newDict = new Dictionary();
  while (n) {
    min = minimum(n);
    newDict.insert(min);
    delete(n, min);
  }
  return newDict;
}
```

> 3. Explain how you can use this dictionary to sort in `O(n log n)` time using only the following abstract operations: insert and in-order traversal

```js
function sort(n) {
  let newDict = new Dictionary();
  for (let i = 0; i < n.length; i++) {
    newDict.insert(i);
  }
  console.log(newDict.inorderTraverse());
  return newDict;
}
```

This is the best I could think of, I actually don't really understand why the book thought this was a good solution. The idea is to copy all of the items into the new list, and then do the sorting by printing out the traversal of all items in-order. This is more congruent with the rest of their sorting algorithms, whereby they insert stuff first, and then handle the sorting part. This isn't really sorting so much as it is taking advantage of using a dictionary like a Binary Search Tree and then traversing by binary search.

## Arrays + Linked Lists = Hash Tables

While dictionaries can be implemented by using arrays or linked lists, we can get significant efficiency improvements by implementing dictionaries using arrays _and_ linked lists. The key to connect those two is to use _hashing_.

A _hashing function_ is just a way to translate something into a numeric representation. There is [no one singular good hash function](https://en.wikipedia.org/wiki/List_of_hash_functions) (e.g. [SHA-256](https://en.wikipedia.org/wiki/SHA-2), [BSD checksum](https://en.wikipedia.org/wiki/BSD_checksum), [Rabin fingerprint](https://en.wikipedia.org/wiki/Rabin_fingerprint)), but what matters is that **it's cheap to calculate and it's evenly distributed**. The number from that function can then be used as the index in an array.

```js
class HashTable {
  ENGLISH_ALPHABET_LENGTH = 26;
  TABLE_SIZE = 500;

  hash(str) {
    let sum = 0;
    for (let c = 0, len = str.length; c < len; c++) {
      sum += Math.pow(ENGLISH_ALPHABET_LENGTH, len-c+1) * char(c);
    }
    return sum % TABLE_SIZE;
  }
}
```

So that's a sample hashing function to translate letters into really big numbers. Why are we making the number super big? So there's a high chance that the key (the word that represents something like an array index for our dictionary) we are translating will be a unique number.

Now we can expand on our `HashTable` to start implementing our dictionary functions!

```js
class HashTable {
  // assume we have the hashing function and constants
  // that we've already implemented

  constructor() {
    this.table = new Array(TABLE_SIZE);
  }

  search(item) {
    let list = this.table[hash(item.key)];
    return list ? list.search(item.value) : null;
  }

  insert(item) {
    let idx = hash(item.key);
    let list = this.table[idx];

    if (list) {
      list.insert(item.value);
    } else {
      list = new LinkedList();
      list.insert(item.value);
      this.table[idx] = list;
    }
  }

  delete(item) {
    let idx = hash(item.key);
    let list = this.table[idx];

    if (list) {
      // delete may have to copy list
      // in the event the linking is broken
      list.delete(item.value);
    }
  }
}
```

Now you can see how the two data structures make a hash table a great implementation of a dictionary! We get the fast search benefits of an `Array`, with the faster inserts/deletes and safety of a `LinkedList` in the event that multiple strings happen to map to the same hashing function. That means **all of these operations are expected to run in constant `O(1)` time, and `O(n)` in the worst-case**, the difference being the event of a hash collision.

If we have no or few collisions, aided by an excellent hashing function, then the linked lists are never filled with more than item, so search and modification is instant. If we have a bad hash function or our luck is bad (_luck_ meaning that we happen to choose words whose hashing functions all result in the same number) then we'll have to operate on just one long linked list which is no different than the `O(n)` runtimes we calculated for lists.

## Practical applications for hash functions and hash tables

With a `HashTable` implementation in our repertoire, we can start to identify use cases when this data structure is good for us to utilize. In general, **hash tables and hashing functions help speed up algorithms that would take `O(n log n)` or `O(n^2)` and turn them into linear time (`O(n)`) algorithms**.

### Pattern matching strings with Rabin–Karp

_Strings_ are an array of characters with the order preserved so we can express things like words in a sentence. We can find patterns in our words (like substrings, palindromes, etc.) using searching algorithms that make use of a hashing function. One such algorithm is _Rabin–Karp_.

```js
function rabinKarp(str, pattern) {
  let hashPattern = hash(pattern);
  let patternLength = pattern.length;

  for (let i = 0, len = str.length - patternLength; i++) {
    let substring = str.substring(i, i+patternLength-1);
    let hashString = hash(substring);

    if (hashString === hashPattern) {
      if (substring === pattern) {
        return i;
      }
    }
  }

  return false;
}
```

The idea here is that we take advantage of the fact that our hash functions return unique but predictable numbers for strings of words. That means that **if we find the same big number inside of our string as the hash number of our pattern, it's probably a match**.

I say "probably" because as we know from our hash table implementation, there is a slight possibility of hash collisions that could generate false positives. Luckily, this is exceedingly rare as the length of our table is large and a prime number (primes of course can only be divided by 1 and themselves, making them harder to have collisions since other numbers can't divide into the same answer).

### Duplication

Since Rabin–Karp helps us detect patterns within a string, we can apply this algorithm to a variety of real-world scenarios that exploit the fact that duplicate hash values indicate that strings are the same:

1. **Unique web pages on the web.** If web pages are just one long document of words and links, then the entire text of the HTML document can be represented as one giant number (via hashing, of course). For crawlers like Google, rather than having to slowly crawl all of the words of a document to verify it is unique, it just has to see if the hash value is in its mega list of hashes. If it isn't we know we have a new web page to add to the crawler.
2. **Plaigarism.** Copying text can be easy to detect because the hashes should be the same, just like crawling web pages. The only problem is adding in a few filler words, some spaces, different formatting… with a few small changes you can create a unique hash and make it impossible to compare one document to the next. Instead, we can leverage pattern matching by scanning the document in chunks and seeing if the vast majority of sub-hashes are the same. This is obviously a downside because it requires significantly more space to store and compare those hashes, but this is still smaller than having to search against every single word betwee the two documents.
3. **Save when changed.** Just as we know that a hash function can numerically represent an entire HTML document, we can hash an entire file with a cryptographic hash to ensure that the document has not been modified by an unauthorized third party. You ever see messages from your laptop warning you that "this third-party application has not been signed?" This is exactly what this is for! Developers send over a hash with their saved changes to prove they were the last ones to touch their programs. They then send those hashes either to app stores or directly to their customers to let them know that their programs haven't been hacked by outside developers, ensuring that their programs can be safely run on other machines.

As you can see, hashing, whether with a table or as a simple function, is a very practical mechanism for dealing with human language data in the form of strings. Strings are the fundamental formal translation of words into arrays, and the hashes are the translators that turn these characters into numbers. This makes dealing with lots of string data very efficient when we hash them out into numeric data.

## The next Daily Problem (4-3)

This round our Daily Problem is back to coming directly from the book, with a preview into our next segment, sorting:

> Take a sequence of `2n` real numbers as input. Design an `O(n log n)` algorithm that partitions the numbers into `n` pairs, with the property that the partition minimizes the maximum sum of a pair. For example, say we are given the numbers `(1,3,5,9)`. The possible partitions are `((1,3),(5,9))`, `((1,5),(3,9))`, and `((1,9),(3,5))`. The pair sums for these partitions are `(4,14)`, `(6,12)`, and `(10,8)`. Thus the third partition has 10 as its maximum sum, which is the minimum over the three partitions.
]]></content:encoded>
      <dc:date>2018-09-13T10:28:00+00:00</dc:date>
    </item>
    <item>
      <title>Dictionaries and Binary Search Trees in JavaScript</title>
      <link>https://www.adamconrad.dev/blog/dictionaries-in-js/</link>
      <description><![CDATA[Dictionaries are so important that we’ve allocated an entire article for them. Dictionaries, also called dynamic sets, are like special arrays. While arrays you can only look up the values by their numeric address, dictionaries allow you to look up items by their values. This gives them tremendous power that we’ll explore today. But first, the answers to the previous article’s Daily Problem.

]]></description>
      <pubDate>Tue, 11 Sep 2018 10:28:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/dictionaries-in-js/</guid>
      <content:encoded><![CDATA[Dictionaries are so important that we've allocated an entire article for them. Dictionaries, also called **dynamic sets**, are like special arrays. While arrays you can only look up the values by their numeric address, dictionaries allow you to **look up items by their values.** This gives them tremendous power that we'll explore today. But first, the answers to the previous article's Daily Problem.

> This article is part of the [Data Structures and Algorithms Series](/tag/algorithms/) following the course [The Analysis of Algorithms](https://www3.cs.stonybrook.edu/~skiena/373/). If you missed the [previous article](/blog/basic-data-structures-in-js/), check that out first as well as [a copy of _The Algorithm Design Manual_](https://www.amazon.com/Algorithm-Design-Manual-Steven-Skiena/dp/1849967202).

## Answers to the Daily Problem

This problem basically asked us to combine what we learned from [Big O Notation](/blog/how-to-analyze-js-functions/) and apply that to one of the fundamental [data structures](/blog/basic-data-structures-in-js/) we saw in the last article.

> For each of the four types of linked lists (singly unsorted, singly sorted, doubly unsorted, doubly sorted), what is the asymptotic worst-case running time for each dynamic-set operation listed?

If you've been following along with this series, this should be pretty easy to figure out.

> a) Search(List, item)

Drawing on our playpen slide tube analogy, we know that we can only enter our `LinkedList` from either ends of the slide. The worst-case is that we enter the wrong end of the slide, and our target is all the way at the other end. So if we have to go through the whole list of `n` items, that means **our worst-case runtime for search is `O(n)` across all 4 kinds of lists**.

> b) Insert(List, item)

One of the benefits we mentioned in the previous article was that insertions and deletions were simpler because we can just tack on stuff to the ends of the list. Since we always have a pointer to the `head` of any kind of linked list, it only requires **constant-time access (`O(1)`) to insert an item onto any unsorted linked list**.

Why does sorting hurt us? Because once the list is sorted, _we can no longer tack on the item to the beginning of the list_. Now the list has some logical ordering behind it, we have to maintain that ordering, which could mean having to scan the entire list to find the exact right place to insert our new item, making **insertions a linear-time runtime (`O(n)`) for sorted linked lists.**

> c) Delete(List, item)

To limit confusion, let's right off the bat say **doubly-linked lists can implement this `O(1)` runtime because they have access to both the previous and next node**. Since you need both to connect them, and thus delete the active `item` by disconnecting it from the list, we're done here.

However, for singly-linked lists, this one is interesting because it depends on the algorithm you want to implement. If you want to delete the item _after_ `item`, this is a straightforward **worst-case for delete is `O(1)`** since you just connect `item.next` to the next of the next.

But if you want to delete the `item` provided, you need to find the previous node, and the only way to do that is to start from the beginning and work your way down until you see that the current node's `next` is equal to the `item` you specified. Only then can you connect `current` to the `next` of the next node, meaning **in the worst-case, deletes can take up to `O(n)` for singly-linked lists**.

> d) Successor(List, item)

This function just asks you to find the next logical item in the list. Logical in this case means that if you have a list that would be represented by an array as `[1,2,3]`, the _successor_ to `2` is `3`. Since logical ordering is based on sort order, this is entirely dependent on if the list is sorted or not, and nothing to do with being singly- or doubly-linked, since both of those implement a `next` property.

**For sorted lists, this is just an access to `next` and runs in `O(1)`, while unsorted lists may require viewing the whole list to finding the logical next item, and thus runs in the worst-case `O(n)`**.

> e) Predecessor(List, item)

Finally, we have something unique to tackle! This function is the first place we see fragmentation between singly- and doubly-linked lists. **For sorted doubly-linked lists, this, too, is `O(1)` because the previous node is a property inherent on the list object, and property access happens in constant time**. This doesn't apply for unsorted doubly-linked lists, because like our quandry with question D, it could take viewing the entire list to find our logical predecessor, and therefore **unsorted doubly-linked lists take `O(n)` to run the predecessor function.**

For singly-linked lists, this presents a problem. If you only ever have access to `next`, how can you possibly know what the `prev` item is? Another way to look at that is to flip the script: rather than looking at the predecessor as the previous node of the current item, we instead want to look for _the previous node's next node_. Yes, see, because we know that the previous node's next node is the same as the current node (`item`) that we're passing in! But to do this we have to start from the very beginning of the list. And if our current `item` is all the way at the end of the list, we'll have to search the entire list of `n` items until we find the predecessor, therefore **singly-linked lists can calculate the predecessor in `O(n)` time in the worst-case, regardless of whether or not they are sorted**.

> f) Minimum(List)

Here we just want to find the minimum item in the list. Whether that's the lowest ordered letter in a list of letters in the alphabet, or the lowest number in the list, we want to find the minimum value. Now it should start to make sense as to why there were 4 list types, one group centered around sorting. When the list is not sorted, you really have no idea where the minimum item could be, so **in the worst-case scenario you have to search the entire list to find the minimum item, which is `O(n)`**. However, if the list is sorted, you know that it is going to be at the very start of the list, which you already have access to in both singly- and doubly-linked lists, and therefore **the minimum value can be found in a sorted linked list in constant `O(1)` time**.

> g) Maximum(List)

Finding the maximum is identical to finding the minimum, but not exactly for the reasons you think (again ,think of the ends of a slide). The obvious part is that **unsorted lists require searching the entire list, so maximum is found in `O(n)` time.** The other obvious connection should be that **sorted doubly-linked lists have access to both the first and last item so the maximum can be found in `O(1)` constant runtime**. So why is **maximum access found in constant time (`O(1)`) for sorted singly-linked lists**? When we add and delete items from a sorted singly-linked lists, we can use that as an opportunity to update a property of the `last` node in the list. By paying that storage cost in those functions, it allows us to have instant access to the last node, which in the case of a sorted list, also happens to be the maximum value as well.

Wow! That's a big _list_ of problems to solve! **As a thought exercise, see if you can't extend this table to include runtimes for sorted and unsorted arrays as well.**

## Thinking about dictionaries

**Think of a data structure dictionary as the eponym of the physical book dictionary is a good analogy.** Now I'm going to give myself some credit here because I think this is pretty clever: I purposefully chose a fancy word like _eponym_ because if you don't know what that means, YOU'RE GOING TO LOOK IT UP IN A DICTIONARY.

And when you look up that word at dictionary.com or whatever your favorite word site is, how do you search for the word? _By its name_. **Dictionaries operate exactly the same way as dictionary books do because searching for content happens by looking up the actual name of the content itself.** Now you know how a `search` method would be implemented.

What about `insert` and `delete`? Well how is a word and its definition laid out in a dictionary that you read?

> **eponym** noun _ep·onym | \ ˈe-pə-ˌnim_ : one for whom or which something is or is believed to be named

The name of the word is the **key** that you've looked up, and the definition of the word is the **value**. So whenever you want to add or remove a word in a dictionary, you _see if the key is available, and if it is, you add/remove the definition_.

As we saw from the Daily Problem, there are some additional (optional) methods you could add to a dictionary as well, such as `min`/`max` for quickly moving to the beginning/end of the dictionary, and `successor`/`predecessor` for moving forward/backward between keys/words (not necessarily forward/backword in memory), very similar to the `next` method we implemented for a linked list. We went into these pretty in-depth at the beginning, so suffice it to say we've thoroughly covered the 7 major methods for implementing dictionaries.

```js
let dictionary = {};

// insert
dictionary["eponym"] = "one for whom or which something is or is believed to be named";

// search
dictionary["eponym"];
// => "one for whom or which something is or is believed to be named"

// delete
dictionary["eponym"] = undefined;
// no different than dictionary["apple"], both are undefined
```

## Improving dictionary methods with Binary Search Trees

**Binary Search Trees (BST)** are a clever data structure that implements all 7 dictionary methods and takes all of the best parts of the dictionary runtimes. The result is a data structure that efficiently supports all of these functions.

BSTs consist of a _root_ or start node and trickle down into two other _child_ nodes: the _left_ and the _right_ child. The _left_ node is always less (in value) than the current root node, and the _right_ node is always greater (in valud) than the current root node. We can also have an optional _parent_ node reference which tracks in the opposite direction. This organization is what allows us to quickly and easily find any value in the tree.

**The best way to think of BSTs is to imagine the [Lady Justice](https://en.wikipedia.org/wiki/Lady_Justice) holding her balance scale with a heavier right-handed side.**

Why the right side? Since the right node is the larger item, if you imagine the tree holding numeric values, and those values represented the mass of the object on the scale, you know that the higher-valued item will weigh more:

```
  6
 / \
4   \
     8
```

If that right there is her scale, then you can clearly see that 8 weighs more than 4, so the scale will tip to the right in favor of the item that weighs more. An elaborate example? Maybe. An example you will remember? _Definitely._

```js
class Node {
  constructor(value) {
    this.value = value;
    this.left = null;
    this.right = null;
    this.parent = null; // optional
  }
}

class BinarySearchTree {
  constructor() {
    this.root = null;
  }

  search(item) {
    let curr = this.root;

    while (item.value !== curr.value) {
      if (item.value < curr.value) {
        curr = curr.left;
      }
      else if (item.value > curr.value) {
        curr = curr.right;
      }
    }

    return curr;
  }

  insert(item) {
    let curr = this.root;

    if (!curr) {
      this.root = item;
      return true;
    }

    while(curr) {
      if (item.value < curr.value) {
        if (curr.left) {
          curr = curr.left;
        } else {
          curr.left = item;
          break;
        }
      }
      else if (item.value > curr.value) {
        if (curr.right) {
          curr = curr.right;
        } else {
          curr.right = item;
          break;
        }
      }
    }

    return true;
  }

  maximum() {
    let max = this.root;

    if (!max.value) return null;

    while (max.right) {
      max = max.right;
    }

    return max.value;
  }

  minimum() {
    let min = this.root;

    if (!min.value) return null;

    while (min.left) {
      min = min.left;
    }

    return min.value;
  }

  preorderTraverse(item) {
    if (item.value) {
      console.log(item.value);
      traverse(item.left);
      traverse(item.right);
    }
  }

  inorderTraverse(item) {
    if (item.value) {
      traverse(item.left);
      console.log(item.value);
      traverse(item.right);
    }
  }

  postorderTraverse(item) {
    if (item.value) {
      traverse(item.left);
      traverse(item.right);
      console.log(item.value);
    }
  }

  traverseTree() {
    inorderTraverse(this.root);
  }
}
```

**Question: Given our practice with Big O Notation, can you figure out the runtime of the methods within this sample `BinarySearchTree` class?**

**Extra Credit: We never filled in the `delete(item)` method for BSTs. To be honest, they're kind of tricky. How do you think you would implement one?**

Would love to see your answers and sample gists [on Twitter](https://twitter.com/theadamconrad).

## Balanced BSTs and the next Daily Problem

While BSTs are very good data structures most of the time, they can still be improved. **_Balanced_ BSTs will automatically rebalance nodes whenever new items are inserted or deleted into the tree.** For example, if you keep adding the natural numbers (1,2,3,4,5…) to a BST, it will be effectively the same thing as a linked list, with the height of the unbalanced tree _`h`_ equal to the length of the items _`n`_:

```
1
 \
  2
   \
    3
     \
      4
       \
        5
```

But if you _balance_ the tree so that we try to maximize as many _left_ and _right_ nodes being used as possible, the end result of a rebalance would look like this:

```
    3
   / \
  2   4
 /     \
1       5
```

This time, the height is halved at every level, and we now know that if we are successively halving something, it's happening in a logarithmic fashion. Therefore **the height of the balanced BST _`h`_ is equal to _`log n`_**.

To get some more practice dealing with this concept for balanced BSTs, the Daily Problem focuses on them:

> You are given the task of reading in `n` numbers and then printing them out in sorted order. Suppose you have access to a balanced dictionary data structure, which supports each of the operations search, insert, delete, minimum, maximum, successor, and predecessor in `O(log n)` time.
>
> 1. Explain how you can use this dictionary to sort in `O(n log n)` time using only the following abstract operations: minimum, successor, insert, search.
>
> 2. Explain how you can use this dictionary to sort in `O(n log n)` time using only the following abstract operations: minimum, insert, delete, search.
>
> 3. Explain how you can use this dictionary to sort in `O(n log n)` time using only the following abstract operations: insert and in-order traversal

Think you've got the answers? We'll find out in two days with the next article! And for more practice, check out these problems from the book:

1. 3-10
2. 3-11
]]></content:encoded>
      <dc:date>2018-09-11T10:28:00+00:00</dc:date>
    </item>
    <item>
      <title>Basic data structures in JavaScript</title>
      <link>https://www.adamconrad.dev/blog/basic-data-structures-in-js/</link>
      <description><![CDATA[Have you wondered how basic data structures like Objects and Arrays work under-the-hood in JavaScript? In this article we’ll be exploring elementary data structures as they are constructued in the JavaScript language.

]]></description>
      <pubDate>Thu, 06 Sep 2018 10:28:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/basic-data-structures-in-js/</guid>
      <content:encoded><![CDATA[Have you wondered how basic data structures like `Object`s and `Array`s work under-the-hood in JavaScript? In this article we'll be exploring elementary data structures as they are constructued in the JavaScript language.

> This article is part of the [Data Structures and Algorithms Series](/tag/algorithms/) following the course [The Analysis of Algorithms](https://www3.cs.stonybrook.edu/~skiena/373/). If you missed the [previous article](/blog/how-to-analyze-js-functions-with-big-o/), check that out first as well as [a copy of _The Algorithm Design Manual_](https://www.amazon.com/Algorithm-Design-Manual-Steven-Skiena/dp/1849967202).

## Answers to the Daily Problem (2-20)

So this one was honestly a bit confusing to me. The question reads:

> Find two functions `f(n)` and `g(n)` that satisfy the following relationship. If no such `f` and `g` exist, write “None.”

The problems are odd because we don't really deal with Little O, which stands for "grows strictly smaller than." Little O is like a stricter version of Big O. This just isn't something you'd have to worry about in the professional world, either as a working developer or in an interview setting. It's also not a topic that is covered very much in an algorithms book. But let's give it a try anyway.

> (a) f(n) = o(g(n)) and f(n) != Θ(g(n))

So this is saying `g(n)` should grow at a smaller rate than `f(n)` should, _and_ should not be the average-case, meaning it can't satisfy both `O(g(n))` or `Ω(g(n))`. Basically, we want an f(n) that is not as bad as g(n), so a simple one would be **`f(n) = n + 1, g(n) = n^2 + 1`**.

> (b) f(n) = Θ(g(n)) and f(n) = o(g(n))

This one should be **none**, and the reason why is that `Θ` is all about being tightly-bound, whereas the little letters (Little O and Little Omega) are all about being loosely-bound. You can't have both, so the answer is none.

> (c) f(n) = Θ(g(n)) and f(n) != O(g(n))

This is also **none**, because the definition of `Θ(g(n))` assumes that you satisfy both the Big O and the Big Omega.

> (d) f(n) = Ω(g(n)) and f(n) != O(g(n))

This one should be fairly straightforward. We want something that satisfies Big Omega and not Big O, meaning we don't want a Big Theta set of functions. **`f(n) = n^2 + 1, g(n) = n + 1`** should satisfy this (essentially, the inverse of problem A).

## Data structures as blocks of memory

One way to group data structures is by how they allocate memory. I will admit, this was not an intuitive way for me to think about elementary data structures, but it's a good thing to keep in mind. There are two major forms of data structures by memory allocation:

### 1. Continuous

Continuous-memory data structures are those that are linked as one block of memory. So when a computer allocates memory in RAM for this structure, it will allocate one block so it can be accessed as one block. The computer won't have to use pointers to find chunks of addresses in order to make sense of all of the data for this one structure. There are lots of data structures of this type we'll cover later, but the only fundamental continuous data structure you need to be aware of is the **array**.

#### Arrays

Arrays are _the_ fundamental continuous data structure that everyone knows. In JavaScript, they have a `typeof === 'object'` but you can distinguish them with `[] instanceof(Array)`.

**Arrays are like [cubbies](https://www.google.com/search?q=cubbies&source=lnms&tbm=isch&sa=X&ved=0ahUKEwj8qP_r1O3dAhWwTt8KHViDD1wQ_AUIDygC) at school.** A cubby is one big piece of furniture with slots to store things like hats, coats, and backpacks.

Like arrays, they store stuff, and they are one object. And like in JavaScript, cubbies can store anything that fits, we don't have any strict types that dictate how our arrays can be allocated like in statically-typed languages.

##### Advantages

* **Constant-time access to elements.** If you provide the _index_, or numeric address inside of the array, we know exactly where the value of that item is without any sort of transformation. That makes array access extremely fast.
* **Space efficiency.** If you only have to take up one block of memory, that's more efficient than if you have to take up lots of discontinuous, fragmented chunks of memory.
* **Colocality.** Building on the previous benefit, because it's one block of memory, that also means that all of the values of the array are located next to each other, making access via traversal also very easy. This allows computers to make use of their caches since we can easily plop the whole array onto (and off) of the cache.

##### Disadvantages

* **Fixed size.** The space efficiency comes at a cost that we can't modify the size of the array later on. If we allocate for 5 items, that's it.

There are two ways to mitigate this: _buffer space_ and _dynamic arrays_. With buffer space, we can leave extra empty room for the array to grow. So let's say you allocate an array for 5 items, well we can buffer in 5 more for a total of 10 allowable items. The obvious downside here is that there is wasted space in anticipation for utilizing arrays for purposes we didn't intend.

The other option is to use dynamic arrays. Dynamic arrays will allocate double the space of the current size of a filled array and copy its contents over to that newly-allocated block. It will then free up the old space that was allocated for the original array. This is like a smart version of buffer space that only adds in the buffer when the array is filled.

It still suffers from the same downside of potentially wasted space, plus the additional operational complexity of allocation and deallocation (`O(n)`). This cost is still `O(n)` because the amount of reallocation work is the convergence of a geometric series (i.e. we double the size half as often because the amount of slots required to fill double with each expansion) which in the end works out to `O(n)`. It also can't guarantee that access will be in constant-time, since the boundary conditions for accessing elements that double the array will now take `O(n)` time.

```js
var myArray = [1,2,3];
```

### 2. Linked

Linked data structures, as you might guess, differ from continuous data structures because they are distinct blocks of memory _linked_ together to form what looks like one cohesive structure. The linking mechanism is called a _pointer_.

A pointer acts as a reference to an address in memory. So while your house is at 742 Evergreen Terrace in Springfield, it's the Evergreen Terrace part that serves as the pointer to your house. In programming, **we make heavy use pointers by passing the value of the pointer around in languages like C and C++.** In JavaScript, however, no such functionality exists.

Instead, **we pass objects by reference in JavaScript**. That means we make a copy of the object as a reference to the original object we want to find, rather than the address we want to use to find it in memory.

In this article we'll cover several of these kinds of fundamental linked data structures, including:

#### Linked lists

I have memories/nightmares of my very early college days doing phone screens for my internships with questions like:

> Tell me how to reverse a singly-linked list in place.

It's funny, looking back now it actually _is_ a good question to ask a sophomore in college because he has no real professional experience to lean on to reply with something snarky like:

> Why would I know how to do that? My list library already has the optimal algorithm.

Anyway back to the task at hand.

I like to **think of linked lists like [those colorful play slides](https://sc02.alicdn.com/kf/HTB1VYtyHpXXXXX3aFXXq6xXFXXXK/200823934/HTB1VYtyHpXXXXX3aFXXq6xXFXXXK.jpg) you see on playgrounds**. There's one way to enter: at the top (or the start of the list). You can't see what's inside the slide/list until you slide down/traverse the list. And each piece of the slide/list is a different color/value. And the only way to get from one section/item to the next is to slide down the slide/iterate to the next time.

And for doubly-linked lists, those are like the crazy kids who would get to the bottom of the slide and dare to climb back up inside to the top. Sure, they're kind of crazy and mildly stupid (the climbers, not the lists), but the reward for the extra work (of climbing or implementing the extra functionality) is increased flexibility and exploration. Now obviously there is more of an implementation cost to doubly-link a list, but this cost is negligible, so it's basically always worth implementing.

##### Advantages

* **Memory is dynamically allocated.** Each time we add a `LinkedList` item to the end of the list, we allocate some more memory. No more and no less than exactly what we need. This is advantageous to arrays because of that space efficiency.
* **Insertion and deletion is simpler.** Since list manipulation operates on the ends of the list, which we have direct access to, these operations are extremely simple, as in, constant-time (`O(1)`) simple .
* **Sorting/rearranging lists is simpler, too.** Since the chain between list items is just a pointer to the next item, we don't have to worry about physically changing the memory space to rearrange the list. All we have to change is who is pointing to who. This is particularly beneficial for really large objects because those objects do not have to move. Moving and finding really large blocks of memory is far more difficult and costly than moving pointers which are of a very tiny fixed size.

##### Disadvantages

* **More overall space.** You not only need space to store your value, but also to point to your other list items.
* **Worse colocality and caching performance.** Since there is no requirement that all of your `LinkedList` items have to be part of the same memory block, it's much more difficult to cache since you could theoretically have to search all of your RAM to find the memory spaces to throw the list into the cache.
* **Random access is slower.** You can't just pluck an item with an index. Remember, this is like a slide. The only way in or out is from the ends, there is a barrier around the middle items of the list. Only from the ends can you make your way to the section of the list that you desire.


```js
class Node {
  constructor(value) {
    this.value = value;
    this.next = null;
  }
}

class LinkedList {
  constructor() {
    this.head = null;
  }

  delete(item) {
    let prev, curr = this.head;
    while (curr.value !== value) {
      prev = curr;
      curr = curr.next;
    }
    prev.next = curr.next;
    curr = null;
  }

  search(value) {
    let curr = this.head;
    while (curr.value !== value) {
      curr = curr.next;
    }
    return curr;
  }

  insert(item) {
    item.next = this.head;
    this.head = item;
  }
}

```

#### Stacks and queues

Once you've grasped linked lists, stacks and queues are pretty easy to understand. **Stacks and queues are just specialized linked lists.**

Technically, both of these can be implemented using a linked list _or_ an array. It is arguably easier to make a queue with a linked list and a stack as an array because of the way the insertion and deletion functions are implemented respectively for arrays and lists.

##### Stack

**A stack is exactly what you think it looks like.** A stack of pancakes. A stack of cards. A stack of whatever.

When you throw pancakes onto a plate, what's the pancake you see at the top? The _last_ one you put on the plate. When you start eating the pancakes, what's the first one you dig your fork into? _Also the last pancake_. This actually has a name: **Last-In-First-Out, or LIFO**.

In the jargon of computer science, really the only difference with a stack and regular old linked list is `insert` is now called `push`, and `delete` is now called `pop`. In fact, you may recognize that because [those are methods on Array.prototype](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array#Methods_2).

That's right, you could actually build a very simple `Stack` class by limiting the scope of an `Array`'s modification to just `push()` and `pop()`.

Why would you need one over an array? Stacks are an ideal data structure for handling depth-first search. Just like you dig deeper into that stack of pancakes, so should you think about using stacks with DFS. We'll get more into search algorithms in a later article.

##### Queue

**A queue is also an aptly-named structure for the thing you do when you wait in line at the deli or the DMV.**

What's that process like? You take a ticket, it has some number on it, and you wait until your number is called. When you are waiting in line, how do they process the orders? From the lowest ticket number to the highest, or in other words, the time people entered the queue. This, too, has a name: **First-In-First-Out, or FIFO**.

Again, this is just a specialized linked list with different names for `insert` and `delete`. Now they are called `enqueue` and `dequeue`. JavaScript has [these methods]([those are functions on Array.prototype](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array#Methods_2) as well, but they are called `push` and `unshift` (or `shift` and `pop`, depending on how you look at it).

No, that's not a typo, `push` is used twice on purpose. With queues, think of it like the tickets: when a number is called from the deli, we're taking the oldest ticket with the lowest number, that's at the _front_ of the queue, or the beginning of the list. When the list was empty, we added to the end of it. As more people are added to the list, we keep adding them to the end of the queue. But when the person behind the deli counter calls our number, where are we in line? _Right in front_, so we don't remove from the end of the list, _we remove from the beginning_.

Why would you need one over a linkedt list? Queues are an ideal data structure for handling breadth-first search. Just like people shuffle across the floor when waiting at the deli counter, so should you think about using queues with BFS. Again, we'll save search algorithms for later.

Okay, I think that's enough for now. Next article we'll cover dictionaries, but right this moment we'll move onto the Daily Problem.

## The next Daily Problem

This one is not in the book, so I'll just transcribe it here:

> For each of the four types of linked lists (singly unsorted, singly sorted, doubly unsorted, doubly sorted), what is the asymptotic worst-case running time for each dynamic-set operation listed?
>
> a) Search(List, item)
>
> b) Insert(List, item)
>
> c) Delete(List, item)
>
> d) Successor(List, item)
>
> e) Predecessor(List, item)
>
> f) Minimum(List)
>
> g) Maximum(List)

Pretty straightforward: for the functions that operate on a linked list, give the Big O time complexity. If you've followed along with this and the previous articles then this question should take very little time to answer. And for more practice, check out these problems from the book:

1. 3-2
2. 3-4
]]></content:encoded>
      <dc:date>2018-09-06T10:28:00+00:00</dc:date>
    </item>
    <item>
      <title>How to analyze JavaScript functions with Big O</title>
      <link>https://www.adamconrad.dev/blog/how-to-analyze-js-functions-with-big-o/</link>
      <description><![CDATA[Welcome back! To review, the previous article explained the basics of asymptotic notation: the upper-bound/worst case, lower-bound/best case, and the tight-bound/average case for how long an algorithm will take to run.

]]></description>
      <pubDate>Tue, 04 Sep 2018 10:28:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/how-to-analyze-js-functions-with-big-o/</guid>
      <content:encoded><![CDATA[Welcome back! To review, the [previous article](/blog/big-o-notation) explained the basics of asymptotic notation: the upper-bound/worst case, lower-bound/best case, and the tight-bound/average case for how long an algorithm will take to run.

As we've mentioned before, the important thing about evaluating algorithms is **correctness** and **efficiency**, and Big O Notation evaluates the efficiency part. To really understand how this works, we'd be best served to look at a few examples and break them down piece-by-piece. But first, onto the Daily Problem…

> This article is part of the [Data Structures and Algorithms Series](/tag/algorithms/) following the course [The Analysis of Algorithms](https://www3.cs.stonybrook.edu/~skiena/373/). If you missed the [previous article](/blog/big-o-notation/), check that out first as well as [a copy of _The Algorithm Design Manual_](https://www.amazon.com/Algorithm-Design-Manual-Steven-Skiena/dp/1849967202).

## Answers to the Daily Problem (2-22)

So for our question today, we're trying to decide which of the asymptotic notations we're working with when comparing two functions:

> For each of the following pairs of functions `f(n)` and `g(n)`, state whether `f(n) = O(g(n))`, `f(n) = Ω(g(n))`, `f(n) = Θ(g(n))`, or none of the above.

It's just a fancy way of saying, "we have a function `f(n)`, is its `g(n)` the best-case, worst-case, or average-case runtime for our function?" Let's take a look:

> (a) `f(n) = n^2 + 3n + 4, g(n)= 6n + 7`

I'm going to offer you the trick I used. As was mentioned previously, we can reduce the `g(n)` down to just the highest-powered variable without the constants. So that means I'm really just looking at whether `n^2 + 3n + 4` is compared to `O(n)`, `Ω(n)`, or `Θ(n)` for this to realistically apply.

For `f(n)`, the worst-case can't be any lower than `n^2`, so that's out. We also know that if worst-case can't work, then neither can the tight-bound theta (since theta requires that _both_ worst and best case can apply). Therefore, we do know the best-case can be no better than `n` given `g(n)` has an `n` in it, so we can safely say that **`f(n) = Ω(g(n))`** is the correct answer here.

> (b) `f(n) = n√n, g(n) = n^2 − n`

This one is a bit trickier, but we can use a neat trick for finding the powers for comparison.

`√n` is really the same thing as `n^0.5`, and when we multiply that with `n`, we're really just saying that `f(n) = n^1.5`. So now we're really just comparing `n^1.5` to `n^2`, which means that `f(n)` clearly has the lower order variable compared to `g(n)`, so **`f(n) = O(g(n))`** is the correct answer.

> (c) `f(n) = 2^n − n^2, g(n) = n^4 + n^2`

Applying the trick here again, we have an exponential function on the left, and a quadratic function on the right. Quadratic functions are lower on the [dominance relation](http://bigocheatsheet.com/), so we can that **`f(n) = Ω(g(n))`** is the correct answer here as well.

## Big O Analysis by example

Now that we've gone over some hypothetic mappings of calculating computational operations, let's actually apply this to real-world algorithms in JavaScript. The most applicable form of Big O Notation in a professional setting would be if you were asked to evaluate the time complexity of the algorithm you've just written on the whiteboard/typed on your computer.

In the following examples, we'll provide the JavaScript code for some algorithm, and then we'll do a Big O analysis line-by-line to show you how we can provide an `f(n) = O(g(n))`.

### Sorting items in an array with Selection Sort

Selection sort is an algorithm that sorts an array by _selecting_ the lowest remaining value and putting it at the end of the sorted portion (the sorted portion is allocated to the front of the array). It looks something like this:

```js
function selectionSort(arr) {
  let min;
  const arrLength = arr.length;

  for (let i = 0; i < arrLength; i++) {
    min = i;

    for (let j = i + 1; j < arrLength; j++) {
      if (arr[j] < arr[min]) min = j;
    }

    [arr[i], arr[min]] = [arr[min], arr[i]];
  }

  return arr;
}
```

If the last line before the brackets seems weird to you, this is how you can [swap variables using ES6 destructuring](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Destructuring_assignment#Swapping_variables).

This is also a great way to illustrate the appropriate uses of `let` and `const`. `let` makes sense for variables like `min`, `i`, and `j` because they are being modified over and over again. `const` makes sense for `arrLength` because the length of the array never needs to change, so we can keep it constant.

So how do we know how much this will cost in terms of Big O time complexity?

Remember the 3 rules from the previous article:

> 1. **Add 1 for simple operations**.
>
> 2. **Add `n` for looping over a set**.
>
> 3. **Whatever your calculations were for 1 and 2, multiply them if the algorithm is called again recursively**.

So let's go through this line-by-line, ignoring lines for whitespace and closing brackets because those are just syntax and are not evaluated:

The first line is the function declaration, and while that requires computation, we're evaluating what is _inside_ of the function, so we can skip this line.

The next two lines are variable declarations, which apply rule 1, and take a constant amount of time (this includes the `length` function on the array). So let's call our total `2` right now.

Now we've hit our first `for` loop. How many elements are we iterating over? From zero up to the length of the array, which we can mark as `n` since that's been the variable we've stuck with for all of our Big O calculations thus far. This brings our total up to `n + 2`.

The next line brings another variable assignment, bringing our total up to `n + 3`.

Now we've hit another `for` loop. But this time it's _inside_ the other `for` loop. How would that work?

Imagine you have 5 cards in your hand, each numbered 1 through 5. Take the first card and add it up against every other card, including itself. How many additions have you made? It should be 5. Now do that for the second card, adding up the other four cards plus itself. The number of additions? Still 6. If you keep repeating this, each card has interacted with each other card, for a total of 25 operations. 25 = 5*5, so if we generalize this for `n` cards, the total is `n*n = n^2`. Therefore, our nested `for` loop is _multiplying_, not adding onto our outer `for` loop.

Since our outer `for` loop thus far has a variable declaration and the loop itself, we're looking at a total of `(n-1)*(n+1) + 2`. Why `n-1`? Because we declare `j = i+1`, so the inner loop will only ever see, at most, 1 less than the entire array.

Next, we have a bunch of things going on all places onto one line. We have a conditional statement, some variable access in the array, and finally a variable assignment (if the condition holds).

This part gets a bit tricky. We know that all of these items add a constant amount of cost in terms of computation. The problem is that with a conditional assignment (like with our `if` statement) the chance of making the condition based on the sum of all of the remaining items in the array to evaluate. That sum is equal to `(n-1) + (n-2) + ... + 2 + 1`, which is approximately equal to `n(n-1) / 2`, bringing the total to `(n-1)*(n+1) + n(n-1)/2 + 2`.

Next, we have our JavaScript destructuring which is acting as a sort of swap function. Since these are just assignments of two variables, we know this is just a constant addition to our operations total within the outer `for` loop, bringing us to `(n-1)*(n+3) + n(n-1)/2 + 2`.

Finally, we return the array to display its newly-sorted self. Referencing a variable is just as costly as assigning one in terms of computation, and now that we're back into the base block of our function, we've completed our cost analysis with a grand total of `(n-1)*(n+3) + n(n-1)/2 + 3`. When we multiply everything out, we get `n^2 + 3n - n - 3 + .5n^2 - .5n`, which simplifies to `1.5n^2 + 1.5n - 3`.

So coming back to the actual asymptotic analysis for O, if we say that `f(n) = 1.5n^2 + 1.5n - 3` and we want to do all of the same simplification techniques when evaluating this to be equal to `O(g(n))`, we're left with the largest power which is just **`O(n^2)`**.

### Matrix multiplication

Did that feel like a lot of work? It should! Because with this next example, we're going to _drastically_ simplify our analysis, even though the algorithm is more complicated.

Matrix multiplication means taking two 2D arrays and multiplying them together to get the multiplied value of all of the cells. If one 2D array is of dimensions `x,y` and the other is `y,z` (they have to have at least 1 dimension in common else you can't actually perform multiplication on the two), we return an array of dimension `x,z` using the following algorithm:

```js
function multiplyMatrices(arr1, arr2) {
  const x = arr1[0].length;
  const y = arr1.length;
  const z = arr2.length;
  let matrix;

  if (y !== arr2[0].length) {
    throw new Error("Matrix cannot be multiplied");
  }

  // initialize matrix
  for (let a = 0; a < x; a++) {
    matrix[a] = [];
  }

  for (let i = 1; i <= x; i++) {
    for (let j = 1; j <= z; j++) {
      matrix[i][j] = 0;

      for (let k = 1; k <= y; k++) {
        matrix[i][j] += arr1[i][k] * arr2[k][j];
      }
    }
  }

  return matrix;
}
```

Much more complex, but with this trick, we can calculate this very quickly: **scan for nesting and multiply**.

The first few lines are constants, so worst-case now is `O(1)`.

Our first `for` loop of length `x`, now we know worst-case is `O(n)`.

We have a 2nd `for` loop where the multiplication starts, but since it's at the same nesting as the first one, we're still at a worst-case of `O(n)`.

The next `for` loop is inside of the other one, and as we saw from earlier, those get multiplied so now our worst-case is `O(n^2)`.

Finally, we have a 3rd `for` loop nested inside the other two, so multiplying this again we now have a **worst-case of `O(n^3)`**. We can skip the `return` statement as we know that's a constant, which we've already determined this far worse than constant time.

Even with all of the initialization, error handling, and matrix math, by simply scanning the function for nesting, we were able to determine very quickly that the time complexity of this function is cubic.

## A bit on why logarithms are special

Remember the example about continuously splitting a deck of cards in half until we reach our target card? This is an example of a logarithm function. **A logarithmic function is the inverse of an exponential function.** That means it tells us "how many times I need to double something to get to some number _n_". Conversely, it also tells us "how many times I need to cut something in half to get down 1."

If you're dealing with a binary search (i.e. choose your own adventure where you either go left or right until you reach a dead end), or you're climbing down the levels of a tree until you hit the leaves, then you know you're dealing with a Big O of `O(log n)`.

The base of the log (meaning base 2 if you have 2 options as in a binary search or base 3 if you have 3 options in a ternary search, and so on) doesn't matter. This is just like how the constant value of a quadratic function doesn't matter in big O analysis. All we care about is the fact that, in general, we're dealing with a logarithmic growth rate for our Big O analysis.

## The next Daily Problem (2-20)

You made it! Big O analysis isn't so bad if you apply some quick scanning tricks. To practice today's work, here's the next Daily Problem:

> Find two functions `f(n)` and `g(n)` that satisfy the following relationship. If no such `f` and `g` exist, write “None.”
>
> (a) f(n) = o(g(n)) and f(n) != Θ(g(n))
>
> (b) f(n) = Θ(g(n)) and f(n) = o(g(n))
>
> (c) f(n) = Θ(g(n)) and f(n) != O(g(n))
>
> (d) f(n) = Ω(g(n)) and f(n) != O(g(n))

## More practice

If you're looking for more [deliberate practice](https://www.redgreencode.com/deliberate-practice-for-software-developers/) with these articles, the _Algorithms_ class we're following includes homework sets and the [first one](http://www3.cs.stonybrook.edu/~skiena/373/hw/hw1.pdf)) just came out today.

Since I'm auditing this course (and so are you by definition if you're reading my articles instead of actually taking the course), I'll include the homework problems that are covered thus far. Once the assignment answer keys are out, only then will I spend any (if at all) time on my answers to the homework problems.

The problems from the homework set that are covered from the first three articles include:

1. 1-17
2. 1-19
3. 1-20
4. 1-22
5. 2-7
6. 2-8
7. 2-19
8. 2-21
9. 2-23
10. 2-24

Happy practicing!
]]></content:encoded>
      <dc:date>2018-09-04T10:28:00+00:00</dc:date>
    </item>
    <item>
      <title>What is Big O Notation?</title>
      <link>https://www.adamconrad.dev/blog/big-o-notation/</link>
      <description><![CDATA[In our previous article we looked at the absolute basics: definitions of data structures and algorithms, how to prove our algorithms are sound, and ways to identify the types of objects we’ll want to use when solving problems.

]]></description>
      <pubDate>Thu, 30 Aug 2018 10:28:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/big-o-notation/</guid>
      <content:encoded><![CDATA[In our [previous article](/blog/how-to-design-an-algorithm) we looked at the absolute basics: definitions of data structures and algorithms, how to prove our algorithms are sound, and ways to identify the types of objects we'll want to use when solving problems.

To recall, we stated that algorithms require both **correctness** and **efficiency** to be sound. We spent a good deal of time talking about how to prove correctness. Today we're going to talk about how to prove efficiency. But first, the Daily Problem…

> This article is part of the [Data Structures and Algorithms Series](/tag/algorithms/) following the course [The Analysis of Algorithms](https://www3.cs.stonybrook.edu/~skiena/373/). If you missed the [previous article](/blog/how-to-design-an-algorithm/), check that out first as well as [a copy of _The Algorithm Design Manual_](https://www.amazon.com/Algorithm-Design-Manual-Steven-Skiena/dp/1849967202).

## Answers to the Daily Problem (1-5)

The question was:

> The _knapsack problem_ is as follows: given a set of integers `S = {s1, s2, . . . , sn}`, and a target number `T`, find a subset of `S` which adds up exactly to `T`. For example, there exists a subset within `S = {1,2,5,9,10}` that adds up to `T = 22` but not `T = 23`.
> Find counterexamples to each of the following algorithms for the knapsack problem.
> That is, give an `S` and `T` where the algorithm does not find a solution which leaves the knapsack completely full, even though a full-knapsack solution exists.

Here's the actual problem, with the answers (and explanations) inlined:

> (a) Put the elements of `S` in the knapsack in left to right order if they fit, i.e. the first-fit algorithm.

**`T = 23; S = {22,21,2}`**. Going from first to last, we add `22`, but at this point there are no other elements that will add up to `23` (even though there does exist a valid solution of `21` and `2`). Our counterexample succeeds!

> (b) Put the elements of `S` in the knapsack from smallest to largest, i.e. the best-fit algorithm.

**`T = 5; S = {1,2,3,5}`**. Going from smallest to largest, we add `1` and then `2`. At this point adding `3` or `5` will put us over `5`, even though obviously choosing just the number `5` would satisfy `T`. Countered again!

> (c) Put the elements of `S` in the knapsack from largest to smallest.

**`T = 23; S = {22,21,2}`**. Same answer as A, and it works here too since they both happen to be ordered from largest to smallest values within `S`.

If any of that doesn't make sense, be sure to [let me know on Twitter](https://twitter.com/theadamconrad).

## Proving an algorithm's efficiency

As we mentioned in the last article, the key to proving efficiency involves **Big O Notation**. What is that exactly? The _O_ specifically refers to the letter commonly used for the name of the function that analyzes an algorithm's _worst case_ (or _upper bound_) on the relative amount of time the algorithm will take to complete (there's also _Ω_ which refers to the _best case_ / _upper bound_ and _Θ_ which refers to the _average case_ / _tight bound_, but in an interview you're only ever going to be quizzed on the worst case, if at all).

I say relative because our _O_ function doesn't return a number such as the number of seconds to complete our algorithm. Instead, it's _relative_ to the number of operations within our algorithm.

## How do we count the number of operations in an algorithm?

We can count operations in an algorithm with 3 simple rules (based on how computer RAM evaluates the cost of computing power):

1. **Add 1 for simple operations**. Adding two numbers (`+`), branching conditionally (`if`), and variable assignment (`var x = 1`).
2. **Add `n` for looping over a set**. If you see `for` or `while` in an algorithm, it's over some `n` number of items, so in the worst case, you need to consider the posssibility that you have to iterate over all `n` items.
3. **Whatever your calculations were for 1 and 2, multiply them if the algorithm is called again recursively**. For example, to calculate the fibonacci sequence on `n` items, each recursive call of fibonacci requires finding the previous value of the fibonacci sequence. If each call requires twice the number of calls, that's exponential (`2^n` specifically) number of calls to fibonacci!

But here's the thing, calculating the _precise_ number of operations for an algorithm is unnecessarily cumbersome. Therefore, **when we refer to _Big O Notation_, we're calculating the time complexity of an algorithm with the multiplicative constants ignored and the lower factors ignored**. So if you found out your algorithm ran in precisely `2n^3 + 4n^2 + 14763n + 1209489` operations, our Big O analysis can reduce this function all the way down to `O(n^3)`.

## Patterns for identifying algorithmic efficiency

So now that we know how to evaluate an algorithm's Big O worst case performance, what do we make of it's efficiency? Let's look at each of the major patterns you'll see for your Big O analysis:

### Constant - O(1)

These are the fastest algorithms because they are comprised of simple operations. Your classic addition function, `function add(x,y) { return x + y }` runs in constant time, it's just the `+` operation! In the real world, these can run on any machine, slow or fast, with essentially any kind of input (provided it actually works), as these will always be able to run efficiently.

### Logarithmic - O(log n)

Imagine you have a deck of cards. Cut the deck in half. Now cut that half of a deck in half. Now cut that portion in half. Now cut _that_ portion… okay you get the idea. If you plotted that on a graph, it would look just like the [`log(n)`](https://en.wikipedia.org/wiki/Logarithm).

If you have an algorithm that is dividing the problem space by some number (like by 2 if you're doing a _binary_ search on a tree), you can bet that algorithm runs in logarithmic time. Obviously this function is now dependent on `n` so it's going to be slower than constant-time functions, but these are so fast on modern machines that you can basically feed it any `n` number of items and it will run just fine.

### Linear - O(n)

Loop over each item in an integer array and print out the number. How many items do you have to print? For an array of 3 numbers, you have to count 3. For 5, it's 5. For `n` it's, well, `n`! This is why we say loops like `for` and `while` evaluate in `O(n)` since we have to touch, at worst, all `n` items.

These algorithms still run pretty fast. Given that CPUs can now run millions of operations per second, it's safe to say a `n` of billions of items is still pretty fast in the real world.

### Superlinear - O(n log n)

How would you get an algorithm like this? Again, imagine a deck of cards. I want you to find a card in the deck quickly. Even worse: I shuffled the deck. But you can find it pretty quickly if you follow these steps: Cut the deck in half until you just have pairs of cards. Sort the cards for each of the pairs (pretty easy to compare if one card is smaller than the other, right?). Now that all of the pairs are set, combine two pairs and repeat the process. Eventually you're going to have the whole deck sorted to easily find your card.

Continuously cutting a deck in half sounds very much like our example where the answer was `O(log n)`.

Looking at the whole deck? Sounds like `O(n)`.

And given that we have to look at all of the cards for each _layer_ that we have to construct, that's `n` number of times we have to make `log n` partitions. Hence, `n log n`! Again, since this is just slightly larger than a linear algorithm, we can run an algorithm like this very quickly on even billions of items.

### Quadratic - O(n^2)

You've got 5 numbers. You need to multiple each number by every number (including itself). So the first number is multiplied 5 times, the second number is multiplied 5 times, the third is too, and the fourth, and the fifth. 5 numbers multiplied 5 times is 25, which is the same is 5^2. So if we generalize this for `n` numbers to multiply by every other number, you've got your `O(n^2)` algorithm.

This is where time needs to start being considered. For tens of thousands of items, we're still moving pretty quickly with generally under a billion operations. Once you get above a million items for this kind of algorithm, we're looking at _trillions_ of total operations to be run. Even your MacBook Pro can't count all of Apple's money in an instant; it's a pretty big number!

### Exponential - O(2^n)

Remember the fibonacci sequence from earlier? A classic example of an exponential algorithm (hint: we can do _much_ better but the naive implementation is exponential). When you have to enumerate all subsets of `n` items, you're dealing with something exponentially painful. And we're talking _real_ painful. With quadratic algorithms we could work with a million numbers. But exponential? Just _40_ items gets us back up to over a trillion. _Oof_.

### Factorial - O(n!)

Arrange a deck of cards from smallest to largest. That's one configuration. Now arrange it from largest to smallest. That's another configuration. Now shuffle the deck. Yet another configuration. Now organize that deck where it's from smallest to largest but order each card where hearts come first, diamonds come second, clubs come third, and spades come last in a given number. These configurations are called _permutations_ of the deck. And in fact finding the number of permutations for _n_ number of items takes `O(n!)` amount of time. So to find all permutations of a standard 52 deck of cards is `52!`.

Do you just how _massive_ `52!` is?

There are more permutations of a deck of cards than there are [_atoms_ on this earth](https://gizmodo.com/there-are-more-ways-to-arrange-a-deck-of-cards-than-ato-1553612843).

You could shuffle a deck of cards every second _since the universe has started_ and still not see every permutation of a deck of cards.

In other words, don't write factorial algorithms. Beyond basically 20 items, it's not only painful, it's essentially _impossible_. Modern computers simply can't handle that number of operations. So if you see your algorithm is pemutating over all of your items, and you have more than a handful of items, you should probably look for another solution to your problem.

## Wrapping up with the next Daily Problem

So now that you understand about Big O Notation and the kinds of bounds you should be striving for, let's put that to the test:

> For each of the following pairs of functions `f(n)` and `g(n)`, state whether `f(n) = O(g(n))`, `f(n) = Ω(g(n))`, `f(n) = Θ(g(n))`, or none of the above.
>
> (a) `f(n) = n^2 + 3n + 4, g(n)= 6n + 7`
>
> (b) `f(n) = n√n, g(n) = n^2 − n`
>
> (c) `f(n) = 2^n − n^2, g(n) = n^4 + n^2`

Think you've got your answers? Good luck, and we'll see you next week!
]]></content:encoded>
      <dc:date>2018-08-30T10:28:00+00:00</dc:date>
    </item>
    <item>
      <title>How to design an algorithm</title>
      <link>https://www.adamconrad.dev/blog/how-to-design-an-algorithm/</link>
      <description><![CDATA[If you missed my previous article, I’m going to spend a series of articles providing notes as I audit Steven Skiena’s CSE 373 Analysis of Algorithms class.

]]></description>
      <pubDate>Tue, 28 Aug 2018 10:28:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/how-to-design-an-algorithm/</guid>
      <content:encoded><![CDATA[If you missed my [previous article](/blog/why-hiring-is-broken-and-how-im-dealing-with-it), I'm going to spend a series of articles providing notes as I audit [Steven Skiena's CSE 373 Analysis of Algorithms class](http://www3.cs.stonybrook.edu/~skiena/373/).

In the first lecture, Skiena mentions you should take a data structures course and a linear algebra course before studying this material.

For professional (read: practical) purposes, we're obviously not starting from scratch, but peaking at the syllabus I do believe we can incorporate data structures by implementing them in JavaScript (this is, after all, a front-end blog) along the way.

And as much as foundational linear algebra will help, there simply won't be anything in a technical interview that would warrant deep study anyway, so we can safely skip this prerequisite.

## What is an algorithm?

An algorithm is **an instruction set.** Kind of like a recipe for a food dish. And just like that recipe for cauliflower rice you found on some random blog, you can translate that recipe into any language you want.

Algorithms are the same way: they are language-agnostic and can be expressed in a human-readable form or machine-readable. The simpler the idea, the easier it's going to be to express in English. The more complex or nuanced the algorithm is, the more likely you'll want to lean on a machine language like Python or JavaScript.

For practical purposes, think of the high-level overview of your algorithm as something you'll explain to an interviewer in English before you dive into the code, but understand that in order to prove your chops as a programmer the code will have to be the primary source for explaining and validating the algorithms you design.

The two defining characteristics of an algorithm that separate an algorithm from other instructions are:

1. **It's correct.** Has anyone ever received credit for implementing an algorithm in a tech interview that didn't produce the correct result every single time?
2. **It's efficient.** We'd like our algorithms to run sometime before we get old.

Now, _technically_, programs don't have to run correctly to be acceptable. Programs that run instructions that are _mostly_ correct are known as _heuristics_. These will become important when studying approximation algorithms later, but for now, assume that correctness is a requirement.

### How do you prove an algorithm is correct?

Proofs were not my strong suit in high school or college. For some reason, it never really clicked for me because the steps in a proof never seemed to line up with the logic my brain used to jump from one step to the next.

Luckily, the easiest way to prove correctness is to prove something **isn't correct.**

Wait, what? How does proving the opposite help us here?

Well, when we're trying to figure out a correct and efficient algorithm to solve a problem, we can narrow the scope of possible choices by eliminating the ones that are demonstratively incorrect. Proving by counterexample can be far easier than other methods.

As a trivial example, suppose we have a total _T = 6_ we want to strive for by adding up numbers in a valid set like _S = [1,2,3]_. It might seem like a very simple algorithm that will solve this problem is to pick numbers from left to right until we reach the total _T_. Even if you add in a big number at the beginning like _S = [5,1,2,3]_ this works since we can just scrap the last two numbers.

What's a counterexample that wouldn't work?

How about _S = [5,2,4]_.

First, we pick 5, which is less than 6. There are no other numbers in our set that could add up to equal 6 after already picking 5, but there _is_ a valid configuration that would still satisfy _T_ (2 and 4). That's proof by counterexample that our algorithm was not correct. It's also a pretty simple counterexample. **Counter-examples can be useful in the real-world to quickly help you assess if the path you're going down is a good one or not.**

If you can come up with a relatively simple counterexample (simple meaning it should only require a handful of variables or items) you know that your algorithm is dead on arrival and you'll need to try something else. If you can't, you're probably on the right track, but that doesn't mean your algorithm is definitively correct.

### What techniques are used to prove correctness?

One way to prove correctness is induction. **Proof by induction indicates that if we can solve for a base case _and_ the general case for `n+1`, we know we've provided a correct answer for all possible inputs.**

There are two important connections to make here about induction which are useful in a professional setting:

1. **Proof by induction is a mathematical form of recursion.** Recursion is a fundamental concept in programming which allows a function to call itself. It allows us to split up large problems into smaller ones.

The classic example here is the Fibonacci sequence (a sequence of integers where the current number is the sum of the previous two numbers). To calculate Fibonacci for a value _n_ in the sequence, you _could_ count up all of the previous numbers manually for each input of _n_, but that would not only be slow and laborious, it would also be difficult to express as a program.

Another way would be to count a few base cases (n = 0 and n = 1) to get the counting started, and then continuously call a `Fibonacci` function with the summed values from the previous step. Recursion is what allows us to accomplish this in code. It is the programming strategy for tackling induction, which is the mathematical strategy for proving statements for algorithms which operate on our sets of data.

2. **Proof by induction is useful for summation.** If you're adding up a lot of inputs together, and you need to prove it will work for all cases, even ones larger than the set you have defined, it can be proven with induction.

But are you ever going to need to formally prove something at work or in an interview? Absolutely not. **But you will need to test your code, and tests are a form of proof.**

So while a formal mathematical proof of induction is likely way more rigorous than you will ever need to showcase in a professional setting, it does set the tone that you can't simply write code and have people assume what you wrote is correct. It needs to be tested somehow, so if you have the mindset that your algorithm needs to be proven correct in some form, you're on the right track to writing quality code.

### And how do you prove something is efficient?

If we'll primarily be using code to express our algorithms, and tests to prove their correctness, Big O notation will be used to prove our algorithms are efficient.

We'll cover Big O in a later post in this series, but the important thing to remember now is that in general, you're going to want to strive for things that take a reasonable amount of time on large data.

For example, if something you design takes an `n!` factorial amount of time, anything over a measly 30 items and you're dealing with numbers larger than the number of stars in the known universe. You _probably_ want something that runs a bit faster than that.

### The big picture on the properties of algorithms

Most CS courses (and most schools) only ever care about these two things. If your teachers and TAs can successfully run your program within a reasonable amount of time, you get an A. Real life doesn't give you an A for these two things because _you don't work in a vacuum_ as you do on a problem set or exam. So what things are missing from the real world picture?

* **Orthogonality:** Is your code dependent on other stuff? Are you writing stateful or functional code? Since most coding whiteboard problems are isolated and self-contained, you usually can't test for this, so make sure you present a portfolio of real-world projects and open source code to demonstrate this
* **Readability:** You can write the hackiest crap to get an algorithm to work, but in the real world other people can't read or use that code, and that's a fail. Make sure if you have time and your code is correct and efficient, to _refactor_ to demonstrate you can write readable, reusable code that is DRY (don't repeat yourself) and orthogonal (or at least that it can be written as part of an orthogonal system)

### The first step in designing an algorithm

So now that we know what defines an algorithm and what is required to prove it's worth using to solve our problems, the next step is to decide how we will design our algorithms. Modeling a problem means knowing the objects you're dealing with, and there are two classes of objects we will cover:

#### Combinatorial objects

A _combinatorial object_ is just a fancy way of saying "what kinds of things can I use to count with?" Since machines are just big 0 and 1 factories, combinatorial objects are the way for us to collect and organize all of the 0 and 1 math our machines are performing thousands upon millions of instructions per second. What kinds of things are we talking about?

* **Permutations:** reorderings of a set. Colloquial terms for this include words like _arrangement_, _tour_, _ordering_, and/or _sequence_.
* **String:** sequence of characters or patterns. Think of strings like permutations but with letters instead of numbers. Words like _text_, _character_, _pattern_, _label_, _sentence_ are key insights that you're dealing with string data.
* **Subsets:** portions of a set. If you see words like _cluster_, _collection_, _committee_, _group_, _packaging_, or _selection_, you've probably got a subset.
* **Points:** locations in space. Words like _node_, _site_, _position_, _record_, or _location_ are all references to _points_.
* **Graphs:** nodes with vertices to connect them and give them direction. We mentioned this much earlier in this article. Words like _network_, _circuit_, _web_, and _relationship_ all describe graphs.
* **Trees:** graphs that flow in one direction and don't end up where they started (acyclic). When they're perfectly balanced, they literally look like a Christmas tree. Words like _hierarchy_, _dominance relationship_, _ancestor/descendent relationship_, _taxonomy_ are all indicators you're dealing with trees in your problem.
* **Polygon:** graphs that do start and end at the same place (cyclic) which represent a shape or region in a physical space. Other words like _configuration_ and _boundary_ all indicate that your problem statement might be dealing with polygons.

#### Recursive objects

A _recursive object_, unlike a combinatorial object like from above, just takes the countable things and asks "what happens when you remove one of those things?"

In other words, if you **remove 1 item from a set it results in a smaller set of permutations.**

Same thing with a string: just delete one letter and you have a smaller string.

Subsets are inherently recursive because **by removing one object, that in itself is a subset**.

With a set of points, split the points down the middle with a line and now you have two smaller sets of points. (Same applies to a polygon).

Deleting one node from a graph is just a smaller graph.

Deleting one node from a tree is just a smaller tree. You get the idea...

With recursive objects, your base case is always the empty or solo set (or in the case of a polygon, a triangle).

With the above objects available at our disposal, we have all of the tools necessary to begin mapping our problem out into real objects which we can then manipulate with our algorithms.

## The Daily Problem

Skiena offers a [problem to think over for each following lecture](http://www3.cs.stonybrook.edu/~skiena/373/hw/daily.pdf). This is sort of like a quick practice problem to test if you've grasped the previous lecture's core ideas. At the start of the next lecture, he goes over the problem (and solution) in detail. So for the August 30th lecture 2 days from now, he's going to **solve Problem 1-5 in _The Algorithm Design Manual_**. Here's what it looks like:

> The _knapsack problem_ is as follows: given a set of integers `S = {s1, s2, . . . , sn}`, and a target number `T`, find a subset of `S` which adds up exactly to `T`. For example, there exists a subset within `S = {1,2,5,9,10}` that adds up to `T = 22` but not `T = 23`.
>
> Find counterexamples to each of the following algorithms for the knapsack problem.
>
> That is, give an `S` and `T` where the algorithm does not find a solution which leaves the knapsack completely full, even though a full-knapsack solution exists.
>
> (a) Put the elements of `S` in the knapsack in left to right order if they fit, i.e. the first-fit algorithm.
>
> (b) Put the elements of `S` in the knapsack from smallest to largest, i.e. the best-fit algorithm.
>
> (c) Put the elements of `S` in the knapsack from largest to smallest.

My plan is to solve the problem before the lecture, throw it up in the next article in the series to start off the article, see if it lines up with the answer given in the lecture, and then compare and contrast the different approaches as well as discuss any flaws or issues that arose along the way.

## See you next time!

Wow, this is probably one of the longest posts I've ever written. I hope this post has convinced you to finally learn data structures and algorithms once and for all. Even if you aren't dedicated to taking the entire course, I hope this stands to be a reference guide if there are ever parts or pieces of this that you might want to reference or check when working with these things in JavaScript.
]]></content:encoded>
      <dc:date>2018-08-28T10:28:00+00:00</dc:date>
    </item>
    <item>
      <title>Why hiring is broken and how I'm dealing with it</title>
      <link>https://www.adamconrad.dev/blog/why-hiring-is-broken-and-how-im-dealing-with-it/</link>
      <description><![CDATA[Programming interviews are a major source of anxiety and angst among the software community. Every few months you see a post on HackerNews saying that we need to seriously rethink how we should be interviewing candidates in the software development profession. The primary reason has to do with how programming interviews are structured today.

]]></description>
      <pubDate>Tue, 28 Aug 2018 10:28:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/why-hiring-is-broken-and-how-im-dealing-with-it/</guid>
      <content:encoded><![CDATA[Programming interviews are a major source of anxiety and angst among the software community. Every few months you see a post on HackerNews saying that we need to seriously rethink how we should be interviewing candidates in the software development profession. The primary reason has to do with how programming interviews are structured today.

## The typical interview process in 2018

If you've ever interviewed at one of the big FAANG (Facebook Amazon Apple Netflix Google) companies you've noticed a trend in how candidates are interviewed, regardless of what kind of position you're interviewing for. The format looks something like this:

* **A recruiter reaches out asking if you want to interview with [FAANG_COMPANY]**
* **30-45 min phone screener**
* **4-5 in-person interviews with a combination of fellow software engineers, engineering managers, and HR**

The real nugget of frustration lies in the type of interview questions they ask. Whether you're an infrastructure engineer or a front-end developer, **these companies love to ask data structure and algorithm questions on a whiteboard**. If you aren't prepared, you can get _rocked_ by them. There are two approaches people seem to take in response:

1. **Interview where this doesn't happen.** Don't want to be subjected to arbitrary interview questions from your CS classes with things you never do anymore? Don't play the game!
2. **Embrace the flawed process and game the system.** In many ways, today's interviews are like studying for the SAT. They aren't an accurate reflection of what you'll actually be doing, but they do have a predictable format that you can study to maximize your chances of success.

But is there a third, better way?

## Why hiring is the way it is today

For a while, I was in the first camp and believed that these tests were contrived, led to too many false negatives (missing good candidates who simply didn't excel in these superficial conditions), and didn't actually measure programming aptitude under real conditions. I've conducted probably over 100 interviews in my career and I will admit I have done both non-whiteboard and whiteboard interview questions with mixed results in both.

But then I realized something.

To preface my hypothesis, I will state right away that this isn't backed up by anything factual since I have never worked at either Facebook or Google. However, given these are highly-desirable companies to work for, I have now come around to understanding why I think hiring is in the tumultuous state it is in:

**Google and Facebook's core businesses revolve around algorithmic problems of large, connected sparse graphs. Therefore, they ask these questions to judge if their candidates can operate on all programming aspects of their business. As a consequence of their success, other companies have followed suit even if their core business is not solving a graph problem.**

Whether it is a network of web pages or a network of people's digital profiles, these businesses, at their core, are about efficiently managing _billions_ of nodes in a connected graph. Problems at this kind of scale cannot be managed without at least some knowledge of graph data structures and algorithms for efficiently interacting with each other.

When you look at it this way, it starts to make sense as to why a company like Google would prefer to ask questions like this. The problem is that other companies have seen Google and Facebook's breakout success and have decided to emulate them without carefully understanding their own business and what is required of their engineers to perform well.

One analogy that resonates with me is when young teenagers read a bodybuilding magazine and decide that if they want to look like Arnold Schwarzenegger, they have to train like him. So the publication outlines Arnold's winning workout that leads him to victory at Mr. Olympia.

Do you see the flaw there? They're showcasing advanced athletes with their advanced workouts. Instead, what you really want to see is **what the successful athletes did as beginners to get to the next stage on their path to ultimate success**.

Put it another way: you don't want to emulate Airbnb's or Stripe's homepage today if you want to look at how to market a successful startup. Instead, you want to look at Airbnb's homepage in 2008, or Stripe's in 2010.

**It is far better to understand success through the _whole journey_, and not just at the end when success has already been attained.**

Getting back to hiring, I think businesses suffer from this same issue, and emulate the FAANG with its current success, rather than look at their earlier hiring practices, or at least take a step back and critically evaluate _why_ they chose to hire the way the did. Perhaps with that insight, we might have businesses who hire to truly reflect the capabilities required of the jobs they are paying for.

## So then why are you doing this?

"Okay", you might say, "so even with all of that said, why are you still in favor of learning data structures and algorithms after being in the business for 10 years?"

Good question! It comes down to four primary reasons:

### If you can do this, you can pretty much handle any interview

As I mentioned earlier, the second camp of folks has embraced that this is our reality in hiring right now. There are [lots](https://www.amazon.com/Elements-Programming-Interviews-Python-Insiders/dp/1537713949/ref=sr_1_1?ie=UTF8&qid=1526172412&sr=8-1&keywords=elements+of+programming+interviews+python) of great [books](https://www.amazon.com/Cracking-Coding-Interview-Programming-Questions/dp/0984782850) that give you the background plus targeted practice to essentially guarantee entry into the coveted FAANG community.

I personally know 4 people in the last year who used one of those two books and put in roughly 80-100 hours of study and landed jobs at Google, Facebook, and Amazon.

This stuff works, plain and simple. Put another way: if you knew that with just 100 hours of work, 2.5 weeks of full-time study, you could _drastically_ increase your chances of earning well into the six figures of income, would you put in the time?

Even if you know that you're "above these games," or that "your work speaks for itself," is it really _that_ much to ask to take what amounts to one semester of college coursework to **have the peace of mind that whether it's Google or some small startup, you'll always pass a tech interview and get a great job?**

Once I looked at it like this, it was hard _not to_ give this a shot, especially considering I never took a dedicated algorithms class in college.

### I enjoy teaching people

One of my favorite jobs of the past few years was acting as an engineering manager. I was fortunate to still spend quite a bit of my time coding, but I was also able to mentor junior engineers. I get as much back as I give when I'm teaching someone something for the first time.

I believe that anyone can do this thing called programming. My hope is that, given the right approach and materials, this can be taught in a practical way that isn't subject to the academic rigidity of a textbook or classroom. To be fair, I will be using a textbook and college course to guide my notes, but my hope is that my interpretation will clear up some of the murky confusion that often arises from overly dense courses.

**If you want a new way of learning the fundamentals and nothing else has worked, maybe this will.**

### Teaching reinforces learning

If you've heard of the renowned [learning how to learn course](https://pt.coursera.org/learn/learning-how-to-learn) you know that the best way to learn something is to teach it to someone else. So yes, as selfish motivation for me to permanently learn this material, I will be employing many of that course's techniques (e.g. chunking, spaced repetition, teaching, alternative notetaking methods) to solidify my understanding of this topic.

### This is part of something even bigger

Lastly, one of my prime motivations for doing all of this is because **I think this is necessary background before one can start to appreciate the fundamentals of front-end UI development.**

Eventually, I'd like to extend this to a second series on the fundamentals of UI frameworks like Angular and React. Both of these frameworks are deeply rooted in understanding how the DOM works.

But before you can truly understand how the DOM works, you need to understand how a tree works. And to understand how a tree works, it helps to understand how data structures like lists and arrays work. When I kept peeling back the layers, I realized it just made more sense to start with the fundamentals of computer science and programming.

## How this is going to work

It is nearing the end of summer and college classes are starting up again. It turns out one of those courses is [Steven Skiena's CSE 373 Analysis of Algorithms class](http://www3.cs.stonybrook.edu/~skiena/373/). Skiena is famous for his [Algorithm Design Manual](http://www.algorist.com/), a book considered by many to be the most approachable introduction to data structures and algorithms. Even Steve Yegge, the famous blogger and Xoogler, [recommends the book](https://steve-yegge.blogspot.com/2008/03/get-that-job-at-google.html) above other canonical texts like _Cracking the Coding Interview_ or the mega textbook, _Algorithms_.

Given I've tried my hand at _The Elements of Programming Interviews_, _Cracking the Coding Interview_, _Algorithms_, and _The Competitive Programmer's Handbook_, all without success, I thought I'd give this one a try. In addition to owning a copy of the book, I figured the combination of the book plus the actual live lectures and course materials might help fill in the gaps from just studying a textbook on your own.

In other words, **I'm auditing Skiena's algorithms course and this series will be my notes as I attempt to learn data structures and algorithms once and for all.** And hopefully, I'll help others along the way.

### Structure

As this is a college course, I imagine many of the exercises and sections are simply not applicable to why you might follow along with this series: to ace a technical interview. So if I see something in the course that you'll likely never see in an interview, I'll acknowledge it was covered, but I won't say too much so we only need to learn exactly what will be useful to us.

#### Each post will follow each lecture

Because I'm going to follow along with CSE 373, I'm going to be taking notes as if I were a student in the course. Those notes will then be transcribed and presented in a teachable format to both reinforce my learning of the material but also to provide a useful filter for my fellow professionals.

#### Each post will cover the publically-available exercises

In case this somehow gets picked up enough to warrant the attention of actual students taking the class, **no material will be published before it becomes publicly available.** This means that if there are homework problems assigned, I will not be publishing my attempts at a solution until the course has published solutions (if ever).

As much as I'd truly love to give you a deep dive into every insight of someone taking the course, I don't think it's fair to the instructor or the students to turn this series into a cheater's reference guide to getting through the course (and hey, I might be wrong on a few things).

#### Each post will provide additional commentary where appropriate

Finally, my hope is to add insight into questions I might have or relevant connections to other material that might help bolster everyone's learning of the material. I certainly won't have all of the answers as I take this course, but my hope is this can be a launching point into more discussion [on Twitter](https://twitter.com/theadamconrad) or other forms of social media so we can figure this all out together.
]]></content:encoded>
      <dc:date>2018-08-28T10:28:00+00:00</dc:date>
    </item>
    <item>
      <title>Mastering the reduce function</title>
      <link>https://www.adamconrad.dev/blog/mastering-the-reduce-function/</link>
      <description><![CDATA[When you first get started learning about functional programming in JavaScript, there are a few functions you grasp pretty easily. Array.prototype.filter is very intuitive, especially if you have worked with a language like SQL which is all about filtering out data. Array.prototype.map is a bit more complex, but you can figure it out pretty quickly if you’re given a few examples.

]]></description>
      <pubDate>Wed, 18 Jul 2018 01:27:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/mastering-the-reduce-function/</guid>
      <content:encoded><![CDATA[When you first get started learning about functional programming in JavaScript, there are a few functions you grasp pretty easily. `Array.prototype.filter` is very intuitive, especially if you have worked with a language like SQL which is all about filtering out data. `Array.prototype.map` is a bit more complex, but you can figure it out pretty quickly if you're given a few examples.

Then there is `Array.prototype.reduce`.

Also known as `fold`, `inject`, `aggregate`, `accumulate`, or `compress` in languages like Haskell, Clojure, and Lisp, reduce is a weird function. For the longest time, I struggled to really grasp how the reduce function works. I think the reason why is because **reduce tries to do a few things at once**. Whereas filter removes items that fail your test, and map modifies your data into a new dataset, reduce kind of does a mishmash of a few things.

![The reduce function can be very confusing](/assets/images/reduce.jpg)

The examples in places like [MDN](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/Reduce) are overly simplistic - they basically show reduce as a way of doing addition by _folding_ one result onto the next. Which is fine, but stumps people the moment you aren't working with numbers. Here's how it usually looks:

```js
// Summation of numbers in an array
[1, 2, 3].reduce((sum, n) => sum + n);
```

This is a perfectly valid example, but you might be wondering: How do you reduce an array of strings? _Why_ would you reduce an array of strings? What are some real-world examples of using reduce to help you in your day-to-day work? We're going to attempt to answer those in this article so you can master the reduce function and finally feel comfortable using it.

## Definition and breakdown

Reduce's job is to take a collection of items and _reduce_ them down to one singular value. The [type signature](https://hackernoon.com/function-type-signatures-in-javascript-5c698c1e9801) of this function looks like this:

```
reduce: (callback: function, initialValue?: any) => any
callback: (accumulator: any, currentValue: any, currentIndex?: number, array?: array) => any
```

We are using [TypeScript's function type annotation syntax](http://2ality.com/2018/04/type-notation-typescript.html#function-types) for this example. In short, it just helps decode the inputs and the outputs. So our function takes in two arguments we've called **`callback`** and **`initialValue`**. What are those arguments?

### `callback` argument and its type signature

**`callback`** is a `function` that takes four arguments itself: **`accumulator`**, **`currentValue`**, **`currentIndex`**, and **`array`**.

**`accumulator`** is of the type of thing you're trying to make your `outputValue`. So if you're reducing an array of `number`s through addition, the `accumulator` is a `number`. It is _accumulating_, or growing, in size **not by the length of the array but by the "growth" of the eventual output value**. In fact, there is an inverted relationship by how the accumulator grows against how the array is reduced. This object is definitely the most confusing, so if this still doesn't make sense, we'll break this down in the later examples.

**`currentValue`** is an item in the array you're trying to reduce. As you iterate over the array, your `currentValue` changes as you move from the first to the last element in that array.

**`currentIndex`** is an _optional_ `number` argument that coincides with the `currentValue` by expressing that value's index. So as you are iterating over your initial array, the `currentIndex` will grow from `0` up to the `n-1` number of elements in your array.

**`array`** is an _optional_ `Array` argument that is the entire input array you provided to the `reduce` function. This might seem like an odd thing to pass in but the key here is that the input array is **not modified**. Functional methods obey _immutability_, meaning they do not modify the array object they are provided. Therefore, regardless of what you do in your `reduce` method, you always have access to your input array because it is never changed throughout the execution of `reduce`.

### `initialValue` argument

**`initialValue`** is an _optional_ argument that provides the initial value for `accumulator` to begin reducing. For example, if you wanted to reduce `[1,2,3]` into the summation of those values, you'd want the `initialValue` to be set to `0` to ensure you are adding against the same type (a `number`) and that this value does not pollute the values that are being manipulated by the array. If you do not set this argument, the `accumulator` will be set to the first value of the array. In the above example, that would be `1`.

### `outputValue` return object

**`outputValue`** is the final form of the `accumulator` object once the entire `array` has been iterated. Unlike most array functional methods, which return arrays, this will return the type that you have built throughout the iteration. This is _not necessarily_ the same type as the values within your input array, as we will see below.

## Examples

Now that we know what goes in and out of the `reduce` function, let's dive into a few examples, step-by-step, in increasing difficulty.

### Easy - sum elements of an array

As we mentioned earlier, the ever-present example of reducing arrays is to sum a list of numbers through addition. It can be succinctly expressed like this:

```js
[1, 2, 3].reduce((sum, n) => sum + n);
```

The easiest way to see how the iteration is working is to throw log statements at each step for our two input arguments:

```js
[1,2,3].reduce((a, c) => {
  console.log(`accumulator: ${a}`);
  console.log(`currentValue: ${c}`);
  return a + c;
});

// FIRST ITERATION
// accumulator:  1
// currentValue: 2
// SECOND ITERATION
// accumulator:  3
// currentValue: 3
// FINAL ITERATION (AS RETURN VALUE)
// 6
```

On the first iteration, we grab the first element in our array, `1`, and set that as our `accumulator` value. We then add our `currentValue`, which is the second element in the array, `2`, onto our `accumulator`, because that is the logic we have decided on for our `callback` function. 1+2 equals 3, so `3` therefore becomes our new `accumulator` value in our second iteration. Since no `initialValue` is set, our second iteration is actually operating on the third and final element, `3`, which is added on to our new `accumulator` value of `3`. 3+3 equals 6, and with no elements left to iterate on in our array, we return the final `accumulator` value of `6`.

Easy! Now, what happens if we operate on another kind of type, like a `string`?

### Medium - simple string builder

One of the most common things I see in JavaScript code involves building strings like this:

```js
const myBuddies = ['James', 'Darien', 'Eric'];
let greeting = 'Greetings friends! ';

for (let i = 0, l = myBuddies.length; i < l; i++) {
  greeting += myBuddies[i] + ' is my buddy. ';
}

greeting += 'These are all of my buddies!';

return greeting;
```

You start with an array and an empty string. You build the string up from the array elements, maybe adding in some additional information, and then you return the new string, fully formed and populated. Now look what happens when I rearrange and rename some variables, do you see the connection?

```js
const array = [some, array, values, ...rest];
const initialValue = 'Greetings friends! ';
let accumulator = typeof initialValue === "undefined" ?
  initialValue :
  array[0];
const manipulation = (a, b) => a.concat(b);

let i = (typeof initialValue === "undefined" ? 0 : 1);
let l = arr.length;
for (i, l; i < l: i++) {
  accumulator = manipulation(accumulator, array[i]);
}

return accumulator;
```

This iterative pattern _is the `reduce` function_. The `callback` within `reduce` is simply some function of `manipulation` of data, iterated across an `array` and saved into some built value called an `accumulator`! The difference is, our last example is 8 lines of code, and to make this in `reduce`, we can express it in a very clean and terse way, with zero local variables needed:

```js
['James', 'Darien', 'Eric']
  .reduce((paragraph, name) => {
    return `${paragraph}${name} is my buddy. `;
  }, 'Greetings friends! ')
  .concat('These are all of my buddies!');
```

This is pretty neat. We were able to remove all local variables and reduce the line count by more than half. We are also able to chain functions together to continue to tack on unique statements at the end of our string builder.

While this instance is a bit more involved and doesn't simply manipulate numbers, at the end of the day, this is still a form of summation, which may seem a bit too simplistic and similar to the first example. Lucky for you, we've got one last example that should be a bit more advanced and is not simply a form of addition.

### Hard - flatten array to dictionary

Our last example is not going to simply be some form of summation. Instead, we're going to build a dictionary (also known as a hash table, which is simply an `Object` in JavaScript) from an array of arrays.

We're going to make the first value in our sub-array be the keys, and the rest of our values in our sub-array be the values. If there are hash collisions (meaning multiple sub-arrays with the same value in the first element), we simply add the remaining sub-array items onto the existing key/value pairing.

How are we possibly going to do this with `reduce`?

We know our initial value is `{}`. We aren't doing a summation (or concatenation if we're dealing with strings), but we are still trying to manipulate our output value, which we know to be an `Object` as a hash table. How can we add things onto a hash table in JavaScript? Square bracket notation! That should be enough to get us what we want:

```js
const arrayOfArrays = [
  ['a', 'ace', 'apple', 'axe'],
  ['b', 'baseball', 'boy', 'bye'],
  ['c', 'cat', 'cold'],
  ['a', 'azure'],
  ['a', 'azure']
];

const dictionary = arrayOfArrays.reduce((dict, page) => {
  const [letter, ...words] = page;

  dict[letter] = dict.hasOwnProperty(letter) ?
    dict[letter].concat(
      words.filter(word => !dict[letter].includes(word))
    ) :
    words;

  return dict;
}, {});

dictionary['a'][0];
// 'ace'
dictionary['a'][dictionary['a'].length - 1];
// 'azure'
dictionary['a'].filter(word => word === 'azure').length
// 1
```

Lots to unpack here. To start, we created our multi-dimensional array called `arrayOfArrays`. Each row in the array starts with the letter of our dictionary. That letter will represent the various pages in our dictionary. You could do this without declaring the local variable, but for the sake of readability within this blog, I chose to separate them.

Next, we construct our actual dictionary. We start with our initial `Object`, a simple empty `{}` hash. Our `callback` represents two things, the `dict` dictionary object that we are constructing, and the `page`, which represents a row in our `arrayOfArrays`. That row has two kinds of strings: the letter representing what class of words we're on, and those words listed on that page.

In languages like Haskell, lists come with functions called `head` and `tail` (also known as `car` and `cdr` in older languages like Lisp). In JavaScript, these don't come out of the box, but thankfully in ES6 we can quickly grab these through the destructuring pattern `[first, ...rest] = array`. We do exactly this to grab our `letter` and corresponding `words`.

Next, we have to build our dictionary. There are two major considerations here: new pages and existing pages.

If we have a new page, we are dealing with the simpler case. All we have to do is populate the page (whose key is `letter`) with our `words`.

If we are dealing with an existing page, we want to add onto that existing page using `concat`. But we can't simply concatenate our list of `words`. What if we've already added a word to our dictionary? That's where our functional cousin `filter` comes in to filter out words that are already included on this page. The filter will return only novel `words`, removing duplicates along the way.

Finally, we test out our new `dictionary` variable to prove our dictionary is complete, can handle adding words late to our prepopulated keys, and will gracefully handle duplicates.

## Final thoughts

Congratulations! You now know how to reduce an array down to a composite value. You've not only mastered the reduce function in JavaScript but in every other language.

As we mentioned in the beginning, the concepts behind `reduce` are the same across languages like Clojure and Haskell, but with different names like `fold` or `inject`. You can also now reason about variants on `reduce`, such as `foldr` (i.e. fold right, which is the same as `reduce`), or `foldl` (i.e. fold left, similar to `reduce` except the iteration happens from the back to the front of the array).

If you're still confused, I've failed you. But fear not; find me on Twitter and I'd be happy to provide even more examples. I will not rest until everyone reading this knows the power of `reduce`!
]]></content:encoded>
      <dc:date>2018-07-18T01:27:00+00:00</dc:date>
    </item>
    <item>
      <title>How to utilize VR on the web with A-Frame</title>
      <link>https://www.adamconrad.dev/blog/how-to-utilize-vr-on-the-web-with-a-frame/</link>
      <description><![CDATA[Reality-altering platforms create new challenges for user interface designers. Devices such as the Oculus Rift and Google Daydream represent the next major evolution in interface computing, and you can get started developing it today, even if you don’t have a fancy headset.

]]></description>
      <pubDate>Fri, 29 Jun 2018 09:25:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/how-to-utilize-vr-on-the-web-with-a-frame/</guid>
      <content:encoded><![CDATA[Reality-altering platforms create new challenges for user interface designers. Devices such as the Oculus Rift and Google Daydream represent the next major evolution in interface computing, and you can get started developing it today, even if you don’t have a fancy headset.

## Getting Started

The first thing you need to know about AR/VR development is what interface you want to develop on. There are lots of hardware and software configurations you can develop for, depending on your application and budget.

## Hardware Considerations

![](https://cdn-images-1.medium.com/max/2000/0*3jDJjY2obTdS8ahm)

The first thing you need to consider is what hardware you’re willing to purchase.

At the high end, dedicated AR/VR hardware usually involves a headset with some sort of glove or controller. These are also so resource-intensive that they require connection to a laptop or ideally a tower, which, in 2018, you may not have anymore, so consider that cost as well if you want to experience VR with the best performance and graphics. The top-of-the-line sets have all essentially converged at the $400 price point, which includes the [Oculus Rift](https://www.oculus.com/rift/), [HTC Vive](https://www.vive.com/us/), and the [Google Daydream](https://vr.google.com/daydream/standalonevr/). If you’re feeling super daring, the [Vive Pro](https://www.vive.com/us/product/vive-pro/) is a cool $800.

The value range gives you a headset to provide a high-fidelity view of VR on your phone. Sets like the [Samsung VR](http://www.samsung.com/global/galaxy/gear-vr/) and [Google Daydream View](https://store.google.com/us/product/google_daydream_view?hl=en-US) will get you quality headgear for your smartphone in the $50–100 range, significant savings from the dedicated sets because the headsets are essentially just goggles rather than powered devices.

Finally, we’ve got the budget headsets which provide the barebones equipment to make VR possible. Sets like [Google Cardboard](https://vr.google.com/cardboard/get-cardboard/) cost as little as $7, and provide the same functionality as your mid-range sets, but with some drawbacks.

Phone-based VR platforms aren’t wearable headsets. These devices act more like goggles, which you have to hold up to your head. From personal experience, I can tell you this gets cumbersome and you’ll find yourself switching hands every few minutes.

The other downside is that the fidelity is low. Seeing things in VR for the first time is cool, even on a Cardboard, but that quickly fades when you are looking at something like GoPro VR video on YouTube. Even though the video may be streaming at 1440 resolution or more, the lenses are not a high enough quality to project that video in a manner that makes it feel real.

## Software Considerations

![](https://cdn-images-1.medium.com/max/2000/0*gunusf9_LNexasiB)

Once you have the hardware purchased, this will dictate the level of software you can develop for.

If you have a self-powered VR set like the Oculus Rift, you can utilize any of the heavy-duty 2D/3D rendering engines like [Unreal](https://www.unrealengine.com/what-is-unreal-engine-4), [Unity](https://unity3d.com/), [Cryengine](https://www.cryengine.com/), [Lumberyard](https://aws.amazon.com/lumberyard/), [Rift](https://developer.oculus.com/pc/), and [Sketchfab](https://sketchfab.com/). These involve some serious programming and graphics knowledge in languages like C# or Python. You’ll also probably need a tower to develop on because the horsepower required for testing and debugging is considerably more when the program is uncompressed and compiled for the desktop environment.

Other VR software is aimed at a more general-purpose audience. These platforms, like [Oculus Go](https://developer.oculus.com/mobile/), [Eon](https://www.eonreality.com/), and [WebVR](http://webvr.info/), all support self-powered sets as well as mobile devices. They don’t require as much power to both develop and test, which is great if you are using your phone as your primary method of experiencing VR.

## Our focus

![](https://cdn-images-1.medium.com/max/2000/0*Xa8YaMPhWtsuz2Vb)

So with all of these platforms and devices to choose from, where do we begin? If money is no object, the Vive and the Oculus Rift are the best-in-class option and will maximize the number of software platforms you can develop on. The downside to going all in is how radically different it is to program for dedicated VR devices. If you don’t have a serious graphics or game development background, it can be pretty overwhelming.

If you are a web or mobile developer and want a nice introduction into VR development with platforms you’re already used to, I’d recommend a wearable like the Daydream View. These can be found for under $50 on Amazon but still provide the headset straps so you don’t have to hold it while you are testing and developing on them.

With the release of the web WebVR platform, you can start developing VR apps with the technologies you’re already familiar with, like CSS and JavaScript. Two of the biggest WebVR frameworks are [A-Frame](https://aframe.io/) (Mozilla) and [React 360](https://facebook.github.io/react-360/) (Facebook/Oculus). In this article, we’re going to focus on A-Frame because of its ease of use, extensive [documentation](https://aframe.io/docs/0.8.0/introduction/), and because it is based on the widely popular [three.js](https://threejs.org/) JavaScript 3D library.

## Installing A-Frame

A-Frame is incredibly easy to install, just add a script tag with the src:

    ”https://aframe.io/releases/0.8.0/aframe.min.js”

That’s it! Insert that into the <head> tag of a standard HTML file and you’re ready to get started! The entire framework is contained within this minified file, just like how you would program a web app with jQuery or React.

If you’re wondering why you insert this script in the top of the file, it’s because the VR environment cannot be loaded without the framework. With most JavaScript source files, it’s beneficial to load the DOM first to prevent the scripts from blocking the content you want to show. In this case, the framework creates its own VR DOM and must be loaded to make use of any of the tags you need to create a VR application on the web.

## Hello (Virtual) World

![](https://cdn-images-1.medium.com/max/2000/0*XKMbJv5AZpGWusxi)

So how do we work with A-Frame? It’s nearly as easy to build an A-Frame application as it is to install it:

    <a-scene>
      <a-text value=”Hello, Virtual World!” geometry=”primitive:plane”>
      </a-text>
    </a-scene>

Yup, that’s really all there is to it. Just insert that into the <body> tag of that same HTML file and you have a WebVR application! You can view our extremely simple demo app [here](https://adamconrad.dev/aframe-demo.html) with a mobile phone using Google Chrome. Impressed yet? Here’s how it works:

The <a-scene> is like the <body> tag or <i-frame> for the WebVR world. It is the container that isolates all of the 3D components and acts as a signal to the device that it needs to render this webpage in VR mode. From there, you can insert a whole set of components called primitives.

For this app, we inserted a simple <a-text> [primitive](https://aframe.io/docs/0.8.0/primitives/a-text.html). We add the properties of our primitive just like our standard HTML tags, using inline attributes. As you might guess, the value attribute tells A-Frame what text to render into the primitive.

Since we are operating in a 3D world, we need to project this text onto something. Now that we are no longer dealing with a static surface like the <body> tag, we need to tell A-Frame how we want to interact with this text primitive. The geometry provides a way to interact with text with something like a cursor by stating how we are projecting the text. In this case, we want to project our text onto another primitive component, the <a-plane>, which is simply referred to as a subset of primitives using :plane.

So how do all of these primitives work together? That’s where the Entity-Component-System architecture (ECS) comes in.

## A Brief Intro on the Entity-Component-System Architecture

![](https://cdn-images-1.medium.com/max/2000/0*K7UyFSMafpQI5FFj)

ECS is the architecture of choice for designing games and 3D environments. This architecture is heavily utilized in Unity, the popular 3D engine gaming framework. A-Frame is an ECS framework built on top of three.js, with specific support for VR devices. So let’s break down each of these words to understand the architecture:

**Entity** refers to house component objects in a scene. These are basically <div>s that are named based on the kind of component you are using. For example, <a-text> is the entity that houses our “Hello world!” text. All A-Frame primitives are some form of entities that combine a subset of components.

**Component **is the reusable chunk of data that gets added to entities to bring them to life. So if <a-text> is the HTML entity we add to our site, the value and geometry components are like the CSS to turn that entity into something meaningful.

**System** is the management system that ties components together across all of the entities they belong to within a scene. So if we again refer to the geometry component, the system for that component is what allows geometry to be shared by <a-text>, <a-triangle>, and <a-circle> primitives, controlling the shapes of the objects and how those shapes appear in space in relation to one another.

Understanding ECS is vital to conceptualizing the A-Frame API because A-Frame is essentially a giant collection of starter entities and components organized by various systems. You can then use these systems to build more complex and engaging components or entities to make your own environments richer and more interactive.

## Next Steps

We’re barely scratching the surface on what VR can do today on the web. Even though VR itself is a brand new and growing platform, truly immersive web experiences are already possible with browsers like [Supermedium](https://supermedium.com/). These VR-first browsers provide a platform to showcase the power of frameworks like A-Frame.

If this article piqued your interest in really diving into WebVR and A-Frame, A-Frame provides a guided tutorial series called [A-Frame School](https://aframe.io/aframe-school/#/1) to walk you through building your first WebVR application.

Once you think you’ve got something unique to share, there are a few places to promote your work. Supermedium offers a [directory](https://webvr.directory/) for compelling WebVR projects, as well as sites such as [WITHIN](https://vr.with.in/).

VR right now is very similar to the early days of the web: fun, expressive, and weird. There’s no need to try to aim for professional or production quality. Rather, we’re in the nascent stage where expression and creativity thrive, pushing people to test the limits of what is capable while the technology is still being fleshed out. There’s no better time to get ahead of the curve with the most bleeding-edge user interface than in AR/VR on the web with A-Frame.

*Originally published at [softwareengineeringdaily.com](https://softwareengineeringdaily.com/2018/06/29/how-to-get-started-in-vr-with-a-frame/) on June 29, 2018.*
]]></content:encoded>
      <dc:date>2018-06-29T09:25:00+00:00</dc:date>
    </item>
    <item>
      <title>What you need to know about WCAG 2.1</title>
      <link>https://www.adamconrad.dev/blog/what-you-need-to-know-about-wcag-21/</link>
      <description><![CDATA[The W3C just released an updated version of the Web Content Accessibility Guidelines a month ago. It’s a pretty massive document with lots of new success criteria and recommendations. Fortunately, you don’t need to read the whole thing to understand what changes you need to make to your websites.

]]></description>
      <pubDate>Wed, 27 Jun 2018 14:11:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/what-you-need-to-know-about-wcag-21/</guid>
      <content:encoded><![CDATA[The W3C just released [an updated version of the Web Content Accessibility Guidelines](https://www.w3.org/TR/WCAG21/) a month ago. It's a pretty massive document with lots of new success criteria and recommendations. Fortunately, you don't need to read the whole thing to understand what changes you need to make to your websites.

Want to jump right to the new success criteria? Here's the full list along with tips on how to pass:

* <a href="#2-1-4">2.1.4 Character Key Shortcuts (A)</a>
* <a href="#2-5-1">2.5.1 Pointer Gestures (A)</a>
* <a href="#2-5-2">2.5.2 Pointer Cancellation (A)</a>
* <a href="#2-5-3">2.5.3 Label in Name (A)</a>
* <a href="#2-5-4">2.5.4 Motion Actuation (A)</a>
* <a href="#1-3-4">1.3.4 Orientation (AA)</a>
* <a href="#1-3-5">1.3.5 Identify Input Purpose (AA)</a>
* <a href="#1-4-10">1.4.10 Reflow (AA)</a>
* <a href="#1-4-11">1.4.11 Non-text Contrast (AA)</a>
* <a href="#1-4-12">1.4.12 Text Spacing (AA)</a>
* <a href="#1-4-13">1.4.13 Content on Hover or Focus (AA)</a>
* <a href="#4-1-3">4.1.3 Status Messages (AA)</a>
* <a href="#1-3-6">1.3.6 Identify Purpose (AAA)</a>
* <a href="#2-2-6">2.2.6 Timeouts (AAA)</a>
* <a href="#2-3-3">2.3.3 Animation from Interactions (AAA)</a>
* <a href="#2-5-5">2.5.5 Target Size (AAA)</a>
* <a href="#2-5-6">2.5.6 Concurrent Input Mechanisms (AAA)</a>

## What do the different conformance levels (A, AA, AAA) mean?
As seen in the WCAG 2.1 document, there are a large number of recommendations for accessibility. It would be unrealistic to expect site owners to adhere to them all, so **each success criterion is given a priority for how important it is to satisfy**.

![Accessibility is important](/assets/images/blur-codes-coding-577585.jpg)

Level A criteria are the most important, followed by AA, and then finally AAA criteria. Each level builds on the other, so it is necessary to satisfy all of the Level A requirements first before trying to satisfy AA or AAA requirements. This is because the W3C [requires full conformance at each level to be recognized as compliant](https://www.w3.org/TR/UNDERSTANDING-WCAG20/conformance.html#uc-conf-req1-head).

> For Level AA conformance, the Web page satisfies **all the Level A and Level AA Success Criteria**, or a Level AA conforming alternate version is provided.

Each section starts with Level A requirements, and builds up to Level AAA, with each requirement adding onto the previous one. For example, Section 3.3 is on input assistance. 3.3.1 and 3.3.2 are Level A requirements. 3.3.3 builds on 3.3.1, which is Level AA. The highest level, 3.3.5 and 3.3.6, base success around much more involved criteria, and thus achieve a rating of Level AAA.

## What conformance levels do I need to pass?
The obvious answer would be to aim for AAA compliance, which is the strictest and most complete standard. The Level AAA standard will create an environment that maximizes your audience reach by providing support for the widest audience of disabled persons.

Practically speaking, **aim for Level AA conformance**. This is the highest realistic conformance level to achieve. The W3C recommends this level as well:

> It is not recommended that Level AAA conformance be required as a general policy for entire sites because it is not possible to satisfy all Level AAA Success Criteria for some content.

One example of Level AAA conformance that cannot be satisfied is [WCAG criterion 2.1.3 for keyboard shortcuts](https://www.w3.org/TR/WCAG21/#keyboard-no-exception). If your application is only available on mobile, such as the Uber app, there is no real keyboard, so it cannot be navigated by keyboard shortcuts, and will not pass this success metric. This, of course, is not a practical requirement because Uber is designed for use on mobile and should not have to try and cater their strategy towards a requirement like this.

![Keyboard events](/assets/images/keyboard-events.jpg)

Now that we have an idea of what level we need to aim for, here are the new success criteria in the updated guidelines and how to pass them.

## New WCAG success criteria

### Level A: The bare minimum

#### <span id="2-1-4">2.1.4 Character Key Shortcuts</span> (<a href="https://www.w3.org/TR/WCAG21/#character-key-shortcuts">source</a>)
##### What it means
Keyboard shortcuts aren't just for power users like vim developers. They are also super useful for the blind, who have no reason for using a monitor or mouse. Therefore, it is a good idea to implement some basic keyboard shortcuts to navigate your site without the need for a mouse.

##### How to pass
* **If you have shortcuts, make sure you can disable them**
* **The shortcut can be mapped to a different key than you provided by default**
* **The shortcut can only be used on a component on in focus.** You can't accidentally trigger the shortcut if it isn't actively on the part of the page that should receive that shortcut.

#### <span id="2-5-1">2.5.1 Pointer Gestures</span> (<a href="https://www.w3.org/TR/WCAG21/#pointer-gestures">source</a>)
##### What it means
Things like touch gestures have many ways of interacting. Single point gestures include tapping or clicking. Multipoint gestures include things like pinch-to-zoom. Path-based gestures include things like swiping or sliding. Your application should account for all of these.

![touch gestures need fallbacks as well](/assets/images/touch-interaction.jpg)

##### How to pass
If you use multipoint or path-based gestures, the **actions that are triggered by those gestures should have a single point fallback**. In other words, if you use the swipe event to mark something as complete, you should also have a touch or click gesture to mark that item complete as well.

#### <span id="2-5-2">2.5.2 Pointer Cancellation</span> (<a href="https://www.w3.org/TR/WCAG21/#pointer-cancellation">source</a>)
##### What it means
Actions can be triggered with, for example, a click. Those actions should be reversible.

##### How to pass
* **Don't use the down-event** (e.g. `keydown` or `mousepress`) unless it is essential
* **Show a popup or alert to undo the previous action**
* **If you use a down-event, an equivalent up-event should be available to undo that action**

#### <span id="2-5-3">2.5.3 Label in Name</span> (<a href="https://www.w3.org/TR/WCAG21/#label-in-name">source</a>)
##### What it means
Images can be used inside of a label to describe content. The component housing those images should have a text fallback of that image content.

##### How to pass
* **If you use an image to label something, the naming attribute should have the text of that image or a description of that image as a fallback.** [Accessible name and description mappings](https://www.w3.org/TR/accname-1.1/#accessible-name-and-description-mapping) include attributes like `alt` for `<img>` tags, [captions, transcriptions, and descriptions](https://webaim.org/techniques/captions/) for `<audio>` & `<video>` content, and `aria-label`, `aria-labelledby`, or `aria-describedby` for all other tags.

#### <span id="2-5-4">2.5.4 Motion Actuation</span> (<a href="https://www.w3.org/TR/WCAG21/#motion-actuation">source</a>)
##### What it means
Actions can be triggered by device motion, such as shaking, tilting, or turning orientation. Motion actions should have the option of being disabled.

##### How to pass
* **Provide a menu item or setting to disable motion events**
* **Don't use motion events unless they are essential to the functionality of the activity**
* **Don't use motion events unless the device is a supported accessibility device with motion in mind** (e.g. haptic response for a Braille reader)

### Level AA: The gold standard

#### <span id="1-3-4">1.3.4 Orientation</span> (<a href="https://www.w3.org/TR/WCAG21/#orientation">source</a>)
##### What it means
Content can be viewed in a variety of orientations, such as landscape or portrait.

##### How to pass
* **Don't limit your content to be viewed in only 1 orientation**

#### <span id="1-3-5">1.3.5 Identify Input Purpose</span> (<a href="https://www.w3.org/TR/WCAG21/#identify-input-purpose">source</a>)
##### What it means
You can programmatically explain the purpose of any input element.

##### How to pass
* **Use available attributes when possible to explain why that input is being used.** For example, use the `autocomplete` attribute on text fields that will take advantage of this functionality.

#### <span id="1-4-10">1.4.10 Reflow</span> (<a href="https://www.w3.org/TR/WCAG21/#reflow">source</a>)
##### What it means
Software like browsers allow you to zoom in and out of the content they display. This causes content to reflow and change how it is laid out on the screen, similar to media queries in responsive design.

##### How to pass
* **Use responsive design meta tags to make the content fit your device.** If content reflows, it shouldn't start requiring you to horizontally scroll.
* **If reflows happen, that should not restrict functionality.** Again, make sure you use the correct meta tags to prevent content from going off of the screen, particularly buttons, links, or other components that provide functionality to the system.

#### <span id="1-4-11">1.4.11 Non-text Contrast</span> (<a href="https://www.w3.org/TR/WCAG21/#non-text-contrast">source</a>)
##### What it means
Contrast exists in all things, not just text. People should be able to interpret and read buttons, select dropdowns, and other components that are not primarily identified as text.

![black and white readers require high contrast](/assets/images/reader.jpg)

##### How to pass
* **Use a [color contrast checker](https://webaim.org/resources/contrastchecker/)** to identify colors that are **greater than a 3:1 ratio of contrast**. For example, use a sufficiently contrasting blue background on a button with white text.

#### <span id="1-4-12">1.4.12 Text Spacing</span> (<a href="https://www.w3.org/TR/WCAG21/#text-spacing">source</a>)
##### What it means
Text needs to breathe. If it's too close together, it's hard to read, especially for those with impaired eyesight.

##### How to pass
* `line-height` > 1.5
* `margin-top`/`margin-bottom` > 2 * font size
* `letter-spacing` > .12 * font size
* `word-spacing` > .16 * font size

#### <span id="1-4-13">1.4.13 Content on Hover or Focus</span> (<a href="https://www.w3.org/TR/WCAG21/#content-on-hover-or-focus">source</a>)
##### What it means
Some consideration is required for content that is only available on hover/focus events, such as popups and tooltips.

##### How to pass
* **You can dismiss this content without having to lose hover or focus.** For example, use the `esc` key to dismiss something.
* **If something is hoverable, you should be able to pass over that content without losing that content.** For example, if you have a tooltip on hover of an acronym like HTML, and you move the mouse onto that tooltip, the tooltip should not disappear because you moved your mouse onto tooltip and off of the acronym.

#### <span id="4-1-3">4.1.3 Status Messages</a> (<a href="https://www.w3.org/TR/WCAG21/#status-messages">source</a>)
##### What it means
A status message is dynamic content like a warning or error. These kinds of messages usually show up at the top of the document and usually break the normal flow of reading the document.

##### How to pass
* **If you issue a status message, an assistive technology like a screen reader should be able to read this message without having to receive focus.** So if you have a blog with a "Read More" button, and the button fails to load the additional content, the status message should be readable by the screen reader without having to use keyboard shortcuts or some maneuver to change where you are in the content flow in order to read what has happened.

### Level AAA: Extra credit

#### <span id="1-3-6">1.3.6 Identify Purpose</span> (<a href="https://www.w3.org/TR/WCAG21/#identify-purpose">source</a>)
##### What it means
Similar to <a href="#1-3-5">1.3.5</a>, you can explain the purpose of all tags in a markup language, not just input elements.

##### How to pass
* Make liberal use of the [ARIA properties](https://www.w3.org/TR/2017/NOTE-wai-aria-practices-1.1-20171214/examples/landmarks/index.html)
* **Provide keyboard shortcuts**
* **Keep pages simple.** Prefer more pages that are simpler in focus than fewer pages with lots of functionality.
* **Use symbols and iconography that users are familiar with** (e.g. right triangle for playing audio or hamburger for menus)

#### <span id="2-2-6">2.2.6 Timeouts</span> (<a href="https://www.w3.org/TR/WCAG21/#timeouts">source</a>)
##### What it means
When a user loses connectivity to a site, such as poor cellular service, popups show things called timeouts which inform the user that service has been interrupted. People like the elderly sometimes cannot complete these kinds of forms in one sitting and may have to revisit this content multiple times.

##### How to pass
* If you use a timeout, **inform the user that there is data loss** and **how long the service will be down for**.
* **Store data for 20 hours or longer.** For example, if you have a checkout form, it should autosave the progress on the form for at least this long.

#### <span id="2-3-3">2.3.3 Animation from Interactions</span> (<a href="https://www.w3.org/TR/WCAG21/#animation-from-interactions">source</a>)
##### What it means
Motion animation can be disabled unless it is essential to the functionality of the component. This is important because high speed or repetitive animations could induce seizures or other physical maladies.

##### How to pass
* Animations that are triggered by some kind of interaction can be disabled unless the animation is required for the component's functionality. **Allow animations to be disabled via a setting or shortcut.**
* **Don't use animations that could cause issues for users.** Police lights, camera flash, repetitive blinking, you get the idea. These kinds of animations are pretty much never needed for an application to work and are just there to add to the design. Avoid them if you can.

#### <span id="2-5-5">2.5.5 Target Size</span> (<a href="https://www.w3.org/TR/WCAG21/#target-size">source</a>)
##### What it means
The target size is the interactable area of an input. It should be sufficiently large enough to make it easy to identify and interact with.

##### How to pass
* **Make the target at least 44x44 pixels.** If the primary target cannot be that large, there should be a fallback component (such as a link) that is that large.
* **Don't worry about links or inline text.** Those are exempt from this rule.
* **Don't worry about elements controlled by the User Agent.** If you can't change the component yourself, this criterion won't hold it against you.
* **Don't worry about elements that are designed to be small on purpose.** For example, an information icon for a tooltip (often portrayed inline as the letter "i" in a circle) is generally presented in a paragraph and would look weird if it were 44x44 pixels in dimension.

#### <span id="2-5-6">2.5.6 Concurrent Input Mechanisms</span> (<a href="https://www.w3.org/TR/WCAG21/#concurrent-input-mechanisms">source</a>)
##### What it means
Content can be interacted with in multiple ways. You can type in a text box, click on the textbox, or touch the textbox with your thumb. Don't assume that users want to interact with those elements in a limited subset of those interactions.

![no input should be turned down](/assets/images/taking-notes.jpg)

##### How to pass
* **Don't restrict any kinds of interactions unless they compromise security or the user specifically disabled them**. For example, if you have a credit card form, just because you've enabled keyboard shortcuts to navigate across input elements doesn't mean you should simultaneously disable touch events because they're a slower interaction.

## You've succeeded!
You should now be well on your way to providing an accessible experience for your users. Obviously, you'll need to provide support for all of the other requirements from the previous versions of the WCAG, but now you are able to update your compliance to the latest standards.

If this all feels like too much for you, [we offer accessibility compliance as a service](https://anonconsulting.com/services/) so you don't have to worry about upgrading to the latest standards. Or [send me a question on Twitter](https://twitter.com/theadamconrad) and I'd be happy to help answer anything that was confusing or difficult to fix.
]]></content:encoded>
      <dc:date>2018-06-27T14:11:00+00:00</dc:date>
    </item>
    <item>
      <title>How to find excellent refactoring opportunities</title>
      <link>https://www.adamconrad.dev/blog/how-to-find-excellent-refactoring-opportunities/</link>
      <description><![CDATA[I was doing some code review today and I realized after the fact that the refactoring I wrote for the review was a great example of how I iteratively evolved a piece of code into something really great. So here’s a step-by-step approach to finding and performing a code refactoring you can be proud of.

]]></description>
      <pubDate>Thu, 14 Jun 2018 21:13:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/how-to-find-excellent-refactoring-opportunities/</guid>
      <content:encoded><![CDATA[I was doing some code review today and I realized after the fact that the refactoring I wrote for the review was a great example of how I iteratively evolved a piece of code into something really great. So here's a step-by-step approach to finding and performing a code refactoring you can be proud of.

# Prerequisite: Read _Refactoring_
The first thing you should do is read Martin Fowler's _Refactoring_. You're super in luck too because the [2nd Edition is going to be in JavaScript](https://martinfowler.com/articles/201803-refactoring-2nd-ed.html).

Since you can't get that now, go read the [Ruby Edition](https://martinfowler.com/books/refactoringRubyEd.html). The Ruby edition is better because it's easier to relate to in the front-end world since Ruby is a simple scripting language and way less verbose than the original version in Java.

Why do you need to read this book? Because it's all tactics. This book has something like 40+ techniques for finding code smells and refactoring them into something beautiful. You probably already knew the really simple ones like Extract Method and Extract Class, but the more advanced ones are where it really shines.

One client I worked with had this heavy reliance on using `switch/case` statements. Those aren't inherently bad but more often than not you can write something cleaner if you [replace the conditional with polymorphism](https://refactoring.com/catalog/replaceConditionalWithPolymorphism.html). Stuff like this is gold, pure gold!

<img class="js-lazy-image" data-src="https://media.giphy.com/media/KdDIGOxbQBAqI/giphy.gif" alt="Refactoring is gold" />

# An example in refactoring
Here's the (obfuscated) code I was working with to begin our journey:

```javascript
const addEvents = (events, label, segment) => events.push([label, segment]);

const main = () => {
  const [upcoming, past] = partition(
    arr, date => today.diff(date, 'day') <= 0
  );

  const events = []

  if (upcoming.length > 0) {
    addEvents(events, 'upcoming stuff', upcoming);
  }

  if (past.length > 0) {
    const byMonth = groupBy(past, date =>
      moment(date).format('YYYY-MM');
    );
    const months = Object.keys(byMonth).sort().reverse();
    months.forEach((month, idx) => addEvents(events, `${idx} months ago`, month);
  }

  return events;
}
```

This code is pretty simple - I've got two arrays of dates: upcoming dates, and past dates (including today). I just need to let the world know what's coming up and what is in the past, so it's not a simple push to an array because each chunk is split off by month (except for the future, which is just labeled future stuff), and each section is really a tuple of label and date.

So what immediately got me tweakin' on this code?

## Single Responsibility Principle
By now the SOLID acronym is engrained in my brain. If you don't know what it is, thoughtbot wrote a [great intro about what SOLID is](https://robots.thoughtbot.com/back-to-basics-solid) so definitely check that out.

The most obvious one that people seem to remember is the first one, Single Responsibility Principle. It's probably the easiest because the definition really says it all. Your class/function/whatever should do one thing and do it well. In this case, I see a bunch of stuff, so my first thought is to break this function up into multiple functions.

How do I know how to break it up? Stuff like `if` statements are literal blocks of code. Since everything is essentially scoped to blocks in JavaScript, anything inside of a bracket is a good candidate to break out into its own function.

I'm also going to cheat and skip a step, because `if` statements also provide another subtle hint: **the `if` keyword can usually be refactored into a guard clause.** Why? Because it is already guarding some code against being run within your function.

If we rewrite the `if` in guard clause notation (meaning, exit early before the function has a chance to run), along with the SRP refactoring, we get:

```javascript
const addEvents = (events, label, segment) => events.push([label, segment]);

const addUpcomingEvents = (events, upcoming) => {
  if (upcoming.length === 0) return;
  addEvents(events, 'upcoming stuff', upcoming);
};

const getPastMonths = past => {
	return groupBy(past, date => moment(date).format('YYYY-MM'));
};

const addPastEvents = (events, past) => {
  if (past.length === 0) return;
  const months = Object.keys(getPastMonths(past)).sort().reverse();
  months.forEach((month, idx) => addEvents(events, `${idx} months ago`, month);
};

const main = () => {
  const [upcoming, past] = partition(
    arr, date => today.diff(date, 'day') <= 0
  );

  const events = [];

	addUpcomingEvents(events, upcoming);
	addPastEvents(events, past);

  return events;
}
```

## Mutation is ugly
Already this looks way better. `main()`'s single responsibility is populating the array of `events` and returning them. Each extracted function has one responsibility based on its function definition. So I guess we can call it a day, right? The next smell I see is this pattern:

```javascript
const myPoorFriendArray = [];

// a world of hurt, warping our pristine, empty friend
blackBoxFunction(myPoorFriendArray);
moreTorture(myPoorFriendArray);
theGauntlet(myPoorFriendArray);

// we chew him up and spit him out, never knowing the horrors he experienced
return myPoorFriendArray;
```

Ya feel me on this one? Our boy `myPoorFriendArray` is now an honorary X-Man: mutated from his original, empty form, into some chimera of various parts of the code. [Kanye said it better than I can](https://www.youtube.com/watch?v=fSJoDuU328k), but this is the stuff I _DON'T LIKE_.

Maybe you come from an OO world like Java and mutation is just a regular, everyday occurrence. If you're woke and you come from a functional background like Elm, you can see that mutants are bad. We don't want no X-Men fighting for us on the coding battlegrounds (sorry Gambit, love you).

So what's the fix here?

1. **Send the empty array in as an argument to our first add function**
2. **The function will take in the empty array and make a copy**
3. **The copy is filled out and returned**

The argument is not mutated, and we remove state dependencies. If you've done any functional programming, terms like _stateless_ and _immutability_ are everyday terms, and that's what we're striving for.

Our stateless code doesn't have [side effects](https://hackernoon.com/code-smell-side-effects-of-death-31c052327b8b), a nasty code smell that makes bugs more difficult to track down because state persists across functions. Without state, what you put into it is what you get out of it each and every time. Making your functions predictable, and thus less likely to include errors you didn't test for.

Here's what this change looks like:

```javascript
const addEvents = (events, label, segment) => events.push([label, segment]);

// we didn't touch the middle functions...

const main = () => {
  const [upcoming, past] = partition(
    arr, date => today.diff(date, 'day') <= 0
  );

  return addPastEvents(addUpcomingEvents([], upcoming), past);
}
```

Now we're getting somewhere! But for the savvy reader, something should still seem off. Can you see what's still wrong?

## Negate the state
**We didn't really remove the state from the add functions.** Yeah, we gave it a starting point in our `main()` function, but the `addEvents()` function is still adding events to a singular `events` array which is getting passed around like a rag doll. How do I know this?

`Array.prototype.push` is a mutable function. It provides an interface to add stuff to an array and returns the number of elements in the enlarged array. How do we do this in an immutable way?

`Array.prototype.concat` is the immutable equivalent of `push()`. It takes an array, combines it with another array, and those two are combined into a newly-allocated array. So we can modify the above slightly to be:

```javascript
const addEvents = (events, label, segment) => {
  if (!label) return events;
  return events.concat([[label, segment]]);
};
```

Now we never touch our inputs. `events` remains as pristine as it was the day it arrived in this humble function, and instead we use `concat()` to combine `events` with the new array we've made, which is the array of our tuple (a point of clarification: tuples don't exist in JavaScript, but I'm using that term to describe our 2-item array to make the distinction clearer here).

Now I'm in audit mode: where else do I need to follow this pattern of not touching the arguments and ensuring I return an `Array` type?

## Provide type safety
If you aren't using TypeScript, this is a great way to practice some proactive type safety to ensure that your function returns objects of the same type regardless of the output. That means don't return an `Array` if you have stuff, but return `undefined` or `null` if you exit early (like in a guard clause). Oh crap, I'm doing that. Looks like another opportunity to refactor!

```javascript
// unchanged but illustrating that all functions, including this one, always return an Array
const addEvents = (events, label, segment) => {
  if (!label) return events;
  return events.concat([[label, segment]]);
};

const addUpcomingEvents = (events, upcoming) => {
  if (upcoming.length === 0) return events;
	return addEvents(events, 'upcoming stuff', upcoming);
};

const getPastMonths = past => {
	return Object.keys(groupBy(past, date => moment(date).format('YYYY-MM'))).sort().reverse();
};

const addPastEvents = (events, past) => {
  if (past.length === 0) return events;
  return getPastMonths(past).reduce((acc, month, idx) => {
	  return addEvents(acc, `${idx} months ago`, month), events);
  };
};
```

Lots to unpack here. So in `addUpcomingEvents()` all we did was make sure we return an `Array` and not `undefined` (ending with just `return;` is shorthand for returning an `undefined` object). We do this because `concat()` returns an array, so we want to make sure all `return` statements provide the same type of object in any given function.

Next I did some refactoring of `getPastMonths()` to handle the sorting an reversing, because the `groupBy` function _technically_ returns an `Object`, and a way for it to return an `Array` is to grab the `Keys` (which is an `Array`) and do our necessary transformations to that array object.

Finally, `addPastEvents()` starts out the same as the upcoming function by ensuring our guard clause returns an `Array` type. The next part is a bit wilder. Originally we were taking an array and iterating over it using `Array.prototype.forEach`. The problem is that this iterator doesn't return an `Array` like we need it to. It simply gives us a platform to view every item in our array.

We also know that in the end, we want one array object that adds in all of the past events. **When you think about needing to combine things into one and return that combined object, think of using [`Array.prototype.reduce`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/Reduce).**

In this case, I knew I needed to add the events (by month) into our `events` array, _and_ return that newly combined array, using `events` as the starting point. The reduce function takes two arguments, a `callback` on how to combine stuff, and an optional `initial` object to begin with.

The `callback` is probably the most confusing part - what is this nested `return` trying to do? The `reduce()` `callback` argument has two arguments of its own: an `accumulator` object, which is initialized to our `initial` object (if you leave it out, it defaults to `undefined`), and the `current` item that is being iterated on. There are two additional optional arguments: the `index` of the item you are iterating on, and the original `array` object you called `reduce()` on. Since we need the `index` argument to label our months, I added that in.

So with all of that said, the `reduce()` function is basically saying:

> For each past month in my array, add it (with its label) onto my accumulated array, which is the cumulation of every previous iteration, starting with my initial `events` array.

## The final result

![Refactored code is good code](/assets/images/blur-close-up-code-546819.jpg)

It was at this point I called it a day and was satisfied with my refactoring for code review I mentioned in the beginning. The final product looks like this:

```javascript
const addEvents = (events, label, segment) => {
  if (!label) return events;
  return events.concat([[label, segment]]);
};

const addUpcomingEvents = (events, upcoming) => {
  if (upcoming.length === 0) return events;
	return addEvents(events, 'upcoming stuff', upcoming);
};

const getPastMonths = past => {
	return Object.keys(groupBy(past, date => moment(date).format('YYYY-MM'))).sort().reverse();
};

const addPastEvents = (events, past) => {
  if (past.length === 0) return events;
  return getPastMonths(past).reduce((acc, month, idx) => {
	  return addEvents(acc, `${idx} months ago`, month), events);
  };
};

const main = () => {
  const [upcoming, past] = partition(arr, date => today.diff(date, 'day') <= 0);
  return addPastEvents(addUpcomingEvents([], upcoming), past);
}
```

To summarize, the major things we covered in this refactoring are:

1. **Simplify functions down to a single responsibility**
2. **Eliminate side effects by using immutable functions and not modifying arguments**
3. **Ensure type safety by verifying all return statements provide the same type of object**

This gives us code that is easier to read, less prone to bugs, and easier to test.

Is there anything else you would refactor? Did I miss something? [Hit me up on Twitter](http://twitter.com/theadamconrad) and let me know what else could make this code better.
]]></content:encoded>
      <dc:date>2018-06-14T21:13:00+00:00</dc:date>
    </item>
    <item>
      <title>If you're really worried about the GitHub acquisition, here's what to look out for</title>
      <link>https://www.adamconrad.dev/blog/if-youre-really-worried-about-the-github-acquisition-heres-what-to-look-out-for/</link>
      <description><![CDATA[Today GitHub was officially acquired by Microsoft. Many of the internet’s citizens are noticeably freaked out about this. I believe this is unnecessary and will attempt to explain why in this essay.

]]></description>
      <pubDate>Mon, 04 Jun 2018 21:13:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/if-youre-really-worried-about-the-github-acquisition-heres-what-to-look-out-for/</guid>
      <content:encoded><![CDATA[Today GitHub was [officially acquired by Microsoft](https://blog.github.com/2018-06-04-github-microsoft/). Many of the internet's citizens are [noticeably freaked out](https://github.com/upend/IF_MS_BUYS_GITHUB_IMMA_OUT) about this. I believe this is unnecessary and will attempt to explain why in this essay.

I believe that developers' concerns boil down to two issues: that Microsoft is evil and will corrupt GitHub, and that the alternatives (primarily GitLab and Bitbucket) are independent, and therefore less/not evil. Once I show that neither issue is true, I'll explain what to look out for when the acquisition is complete and the terms of service are updated to reflect the policies that Microsoft would like to enact.

Full disclosure: I used to work for Microsoft. This was nearly a decade ago, and I have no idea what goes on behind their walls. All of my connections who worked there now work somewhere else. I figure I should say this now because someone will try to be an internet sleuth and attempt to invalidate my arguments because of that connection. I don't think it plays a role, but you'll have to judge for yourself.

# Myth: Microsoft is evil and will corrupt GitHub

This is the primary reaction I see from the development community. Developers have long hated Microsoft, and this mostly stems from the fact that Steve Ballmer did not know his customers (specifically, his customers who were developers). For a long time, Microsoft did not seem to resonate with developers because they made quite a few flaws: lack of open source involvement, slow or laggard IE development cycles, and tooling lock-in, for starters.

Once Microsoft ousted Ballmer in favor of Nadella, this all changed. Microsoft [joined the Open Source Initiative](https://opensource.org/node/901). They also joined [The Linux Foundation](https://www.zdnet.com/article/microsoft-completes-its-linux-journey-joins-the-linux-foundation/). And of all companies, GitHub admitted that Microsoft was the [#1 contributor to open source on that platform in 2017](https://octoverse.github.com/).

And this isn't even about making peace, this is about money. And money talks:

![Microsoft is better with open source](/assets/images/msft-better-w-open-source.png)

Note that the cursor in the screenshot is pointed right when Nadella took over. When they corrected their ship, this made a _huge_ impact on their bottom line. In fact, it's worked so well that they are now once again [bigger than Alphabet](https://www.cnbc.com/2018/05/29/microsoft-passes-alphabet-by-market-cap.html). What is the bottom line?

**Microsoft is financially incentivized to not screw this up.**

They've already invested so much into open source. Let's say, worst case scenario, they were to turn their back on OSS and decide to introduce some truly draconian terms for GitHub. History has shown those kinds of moves completely stall their stock. This is what Ballmer did and the company suffered financially. They don't want to repeat the same mistake twice.

## Don't forget LinkedIn

Microsoft has been involved with a social community before. [Nearly two years ago to the day they acquired LinkedIn for nearly x4 more](https://news.microsoft.com/2016/06/13/microsoft-to-acquire-linkedin/). You know what happened?

Not all that much.

In fact, the change in terms of service was so inconsequential that [LinkedIn noted](https://blog.linkedin.com/2017/april/18/updates-to-our-terms-of-service) that they would not be influenced by Microsoft under blanket terms of service:

> You might have heard that at the end of last year we became a part of Microsoft, but our services are still provided under a separate Terms of Service. We’re excited about leveraging Microsoft technology and resources to offer more valuable features and services to you.

And I've been on LinkedIn for the last 2 years. I'm neither frustrated nor thrilled. LinkedIn is just as much of a recruiter Zombieland as it has always been. You don't have to trust Microsoft when they say they don't want to change GitHub, but they are not offering any indication that they want to mess with it.

# Myth: GitLab and Bitbucket are safer options

The other issue I see is that if there is an unwavering hatred for Microsoft, that we must all leave for better pastures. The two biggest players in the space that are still independent are GitLab and Bitbucket. Are they _really_ all that much better than GitHub?

## They're all for-profit businesses

The most remarkable thing to me in reading the public's reaction is that people make it seem like GitHub was an open source non-profit company. That because they were the premiere company to publish your open source code, it must mean they were pure in intentions.

They're very much a for-profit and primarily make money off of storing private repositories. GitLab and Atlassian are no different! In fact, now that GitHub is part of Microsoft, they're a public company just like Atlassian.

Even though the business models for each are somewhat different in how they make money off of the code storage, the fact is each of those companies is closed source, with specific legal terms to designate what is owned by the contributors, and what is owned by the storage company. And they're _all_ trying to make money (and there's nothing wrong with that).

## You can still deploy your code to mega-corps

Even if you go with GitLab or Bitbucket, where are you going to deploy your code? Because unless you plan on buying your own servers and scaling like we did in the 90s and 2000s, you're going to use one of a few big players:

1. DigitalOcean. A company whose debt is mostly [financed by megabanks](https://www.crunchbase.com/organization/digitalocean#section-locked-charts) like HSBC and Barclays.
2. Heroku. A company now owned by Salesforce, a publicly traded company worth nearly $100BB.
3. Amazon Web Services. A company with a bigger market cap than Microsoft.
4. Azure. Which is owned by you know who...

So even if you self-host your code for the sake of using git with your team, you're still handing over your code to another massive company when you go to turn the lights on for customers. And you may have already done that deed with Microsoft.

## Look at the terms

The GitHub terms of service only mention open source [in section D.6](https://help.github.com/articles/github-terms-of-service/#d-user-generated-content):

>  This is widely accepted as the norm in the open-source community; it's commonly referred to by the shorthand "inbound=outbound". We're just making it explicit.

This isn't even in the ownership section, which clearly states:

> You retain ownership of and responsibility for Your Content...we need you to grant us — and other GitHub Users — certain legal permissions, listed in Sections D.4 — D.7. These license grants apply to Your Content. If you upload Content that already comes with a license granting GitHub the permissions we need to run our Service, no additional license is required. You understand that you will not receive any payment for any of the rights granted in Sections D.4 — D.7. The licenses you grant to us will end when you remove Your Content from our servers unless other Users have forked it.

You can continue to read the rest of those sections, but the real key nugget that you're probably worried about is the legal permissions that apply to you:

> We need the legal right to do things like host Your Content, publish it, and share it. You grant us and our legal successors the right to store, parse, and display Your Content, and make incidental copies as necessary to render the Website and provide the Service. This includes the right to do things like copy it to our database and make backups; show it to you and other users; parse it into a search index or otherwise analyze it on our servers; share it with other users; and perform it, in case Your Content is something like music or video.
>
> **This license does not grant GitHub the right to sell Your Content or otherwise distribute or use it outside of our provision of the Service.**

The rest of those sections simply state that if you post public code it can be viewed by anyone, if you contribute to a repo it's bound to the repo's declared license, and when you upload code it comes with "moral rights," meaning you have the right to seek attribution and credit for the code you have published.

So how does this compare to ownership and IP for GitLab and Atlassian? [For GitLab](https://about.gitlab.com/terms/#subscription):

> 4.3 Customer and its licensors shall (and Customer hereby represents and warrants that they do) have and retain all right, title and interest (including, without limitation, sole ownership of) all software, information, content and data provided by or on behalf of Customer or made available or otherwise distributed through use of the Licensed Materials (“Content”) and the intellectual property rights with respect to that Content.

And [for Atlassian](https://www.atlassian.com/legal/customer-agreement):

> 7.4 Your Data. “Your Data” means any data, content, code, video, images or other materials of any type that you upload, submit or otherwise transmit to or through Hosted Services. You will retain all right, title and interest in and to Your Data in the form provided to Atlassian. Subject to the terms of this Agreement, you hereby grant to Atlassian a non-exclusive, worldwide, royalty-free right to (a) collect, use, copy, store, transmit, modify and create derivative works of Your Data, in each case solely to the extent necessary to provide the applicable Hosted Service to you and (b) for Hosted Services that enable you to share Your Data or interact with other people, to distribute and publicly perform and display Your Data as you (or your Authorized Users) direct or enable through the Hosted Service. Atlassian may also access your account or instance in order to respond to your support requests.

**TL;DR: you own your code.** In fact, if there was one company in that three that I am worried about, it's Atlassian. **Atlassian can collect, copy, store, modify, and use your data.** This isn't spelled out with GitLab or GitHub. So don't think the alternatives are any better for you now that Microsoft owns GitHub.

# Assuming it's all good, what can go wrong?

So if you're convinced that you don't need to jump ship, the truth is things _could_ go wrong. Here are the things to look for when the deal ultimately goes through and the acquisition is 100% complete:

1. **Look out for a GitHub conversion announcement.** Just because they've announced an acquisition does not mean the services have completely rolled over to Microsoft. In fact, the current terms of service for GitHub are dated May 25th which is pre-announcement. If things change and either GitHub or Microsoft announces the rollover is complete, make sure to read the changes carefully. If they change their terms, they'll make you accept or reject the new terms.
2. **If they introduce new terms, see who owns them.** As I mentioned earlier in LinkedIn's conversion announcement, they stated that they still owned their own separate terms of service. This will likely be the case for GitHub as well. If that isn't the case, you'll have to read Microsoft's terms as well as GitHub's. Like class inheritance, whatever you read on GitHub is layered on top of what Microsoft is applying to their software licenses.
3. **If the terms change, specifically check Sections D.3-D.7 for change in ownership or rights.** That's the core section that deals with who owns and distributes the code. If this becomes more restrictive or unfavorable to your community, that would be a good time to re-evaluate your source code hosting strategy.

Is this acquisition a big deal? Yes. Is it a bad deal for you? Probably not. Keep calm and code on, but if the terms change, read them carefully and re-evaluate if they are no longer favorable to your interests.
]]></content:encoded>
      <dc:date>2018-06-04T21:13:00+00:00</dc:date>
    </item>
    <item>
      <title>How to Improve on Naming Contexts in Domain-Driven Design</title>
      <link>https://www.adamconrad.dev/blog/how-to-improve-on-naming-contexts-in-domain-driven-design/</link>
      <description><![CDATA[I’ve been using Elixir’s Phoenix framework for a few years but most of my projects were on the older MVC style. Phoenix 1.3 introduced the concept of contexts (commonly known as bounded contexts). This changed the way you architect your applications.

]]></description>
      <pubDate>Wed, 30 May 2018 23:25:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/how-to-improve-on-naming-contexts-in-domain-driven-design/</guid>
      <content:encoded><![CDATA[I've been using [Elixir's Phoenix framework](https://phoenixframework.org) for a few years but most of my projects were on the older MVC style. Phoenix 1.3 introduced the concept of contexts (commonly known as bounded contexts). This changed the way you architect your applications.

I was _really_ confused by this at first (naming is, after all, is [one of the hardest things in programming](https://skeptics.stackexchange.com/a/39178)). After a bunch of exploration, I've got a surefire way to improve how you name these things.

## What are bounded contexts, exactly?

A [bounded context](https://martinfowler.com/bliki/BoundedContext.html) is a conceptual grouping of a set of data models. The conceptual piece is key here because it means _you_ decide on how you want to split up your models. This freedom is also a crutch because it also means there is no right or wrong way to create a context. For example, the name of these two contexts...

![Bounded context example](/assets/images/bounded-context-intro.png)

...is just as valid as these...

![Bad bounded context example](/assets/images/bounded-context-intro-alt.png)

But hopefully, this will post will show you how to get closer to the first example and further from the second.

## What is domain-driven design, and how does it relate to bounded contexts?

Now that we know what a bounded context is, it makes sense to define [domain-driven design (DDD)](https://airbrake.io/blog/software-design/domain-driven-design) as well. DDD is the practice of **managing complexity of software applications by relating their underlying data models to domain logic**. That's a mouthful; can we break it down further?

The **domain** is an [ontology](https://en.wikipedia.org/wiki/Ontology_(information_science)), meaning how things are grouped in the world. For example, the word _anesthesia_ has a very specific connection to the domain of medicine. Another word, like _Mike_, can belong to multiple domains, such as the domain of Biblical names, or in the domain of politics as it relates to the NATO alphabet.

When the design is **domain-driven**, it means we place the **model** of our domain (e.g. a `PlayingCard` in the domain of `Poker`) in a **context** (e.g. the contextual grouping, such as a `Game`) to help manage the complexity.

![The elements of domain-driven design](/assets/images/domain-driven-design.png)

## Can't I just have one context and call it a day?

**Yes!** In fact, that is preferred for very small applications. What do you name your one and only context for your small application? The application! If you're struggling with naming and you're just getting started with say, a script with one model, **the context of your application can be named _your application_**. So if we took the above example and gave it a real context and model, it could very well look like this:

![Simple domain-driven design example](/assets/images/simple-ddd.png)

## So what happens when my app becomes huge?

This is where DDD is useful. Domain-driven design is specifically for handling the complexity of growing applications as they add more and more models. It doesn't _really_ make sense for an app with one model. **Once you get to about 4 models, that's a good time to start looking at binding your models to multiple contexts**.  This isn't a hard-and-fast rule, so don't feel like you _have to_ break out into multiple contexts, but once you get above 4 models, those contextual groupings will begin to surface.

## Naming contexts by example

I've always felt the best way to figure things out is to learn by example. I'm going to illustrate 3 extremely simplified versions of popular applications, and we'll reveal how we can organize the data models of these applications into contexts. The main thing to remember is that **naming contexts boils down to how you would describe the set of models to a friend in one word**.

### Example 1: Quora

Quora is a forum where users of the site can have their questions answered by the community. Questions can be upvoted by the community that are deemed intelligent or thought-provoking, helping it gain a larger audience for people to participate. Similarly, popular answers can be upvoted to help bring the best answers to the top, logically grouping the best question with the answer.

![Quora domain modeling example](/assets/images/quora-no-ddd.png)

Looking at this setup a natural hierarchy emerges. Is there a signal here that we can start creating separate contexts? The first thing I see is that `Question` and `Answer` operate on the same level - they have equal precedence in the total domain hierarchy of this application. Now taking my advice from above, I have to ask myself:

> How would I describe `Question` and `Answer` to a friend in one word?

First of all, we can look at the description I gave in the beginning. In addition, we remember that contexts are nouns, so when I'm parsing those first descriptive sentences, I see words like **forum** and **site**.

Another technique is to bring these models into reality. Where do you see Q&A in the real world? At tech talks, for example, it's usually at the end of a presentation where the speaker has a discussion with an audience. Using the noun technique in the previous sentence, potential context ideas are **presentation** and **discussion**. Now we can apply the domain-driven part of our design to ask ourselves:

> What domain does this belong to, and are there any conflicts or ambiguities we need to worry about?

For presentation, this belongs to the domain of things like public speaking and business. The problem is, presentations are also conflated with the MVP (Model-View-Presenter) pattern. It is also a specific format for PowerPoint. Given that this word has specific connotations with programming jargon, we can rule this one out. That leaves us with **discussion** and **forum** as our context for our Q&A models. Honestly, either will work here, but one hint is that forum relates more to this particular domain than discussion does, so I think we'll go with forum.

Now we can repeat the process for `User` and `Organization`. Since these aren't on the same level, we could argue that they shouldn't be in the same context, but we have to give them _a_ context because there really isn't a point in making a 1-to-1 mapping of models to contexts for an application this small. But we also see a natural mapping here: people belong to organizations all of the time, whether it's a company or a volunteering outfit.

How do we describe people in an organization? We say they're organized by some hierarchy and are usually grouped into teams or divisions. That gives me a few nouns: **hierarchy**, **grouping**, **team**, and **division**. I can already see that team and division are immediately out since that is an example of how organizations are split and don't fully encompass the user-organization relationship. Grouping is good, but the word *group* itself does creep a bit too close to computer terms such as `GROUP BY` or the C# `grouping` keyword. So I'll go with **hierarchy**.

![Quora domain modeling example with contexts](/assets/images/quora-ddd.png)

Nice work! We have a clear designation between two contexts that make sense. One thing to point out is that **contexts can be renamed or changed later**. It's not important that we anticipate the use of these contexts as we add models. Sure, `Hierarchy` might limit our flexibility, as it could be a stretch to add in helper models like `Address` into this context, but you can worry about refactoring later.

### Example 2: Twitter

Twitter is a microblogging service that allows people to express their thoughts in a very limited space. All messages (called "tweets") have a 280 character limit and are broadcast to followers. Followers can then interact with their connection's messages by liking or sharing (via "retweeting") content that they think is valuable to their own networks.

![Twitter domain modeling example](/assets/images/twitter-no-ddd.png)

What did you come up with?

Let's again apply the same heuristics from above to decide what our contexts will be. The first thing that stands out to me is the intimate tie between `User` and `Tweet`. It's the absolute core of this platform. Simplifying the noun technique, my first thought was that this _is_ the **microblog**. However, given that Twitter has increased its word count from 140 to 280 characters, I feel like the *micro* portion of this is basically irrelevant, but the **blog** portion aptly contextualizes this relationship. Plus, it's a natural context for future potential models like `Comment`.

Next, there are the user-related interactions: following/followers is largely abstracted away since it's a `User` to `User` relationship. `Like` and `Share` are new models, and do have similar mechanisms that feel like they could be grouped. What did I just call these things? **Interaction**, that's one (in fact, I also used it in the opening paragraph). I also used **network** which would describe the following/follower relationship, but again, networking is computer jargon and that could collide with a lot of other concepts we are working with. I think **interaction** is a good one:

![Twitter domain modeling example with contexts](/assets/images/twitter-ddd.png)

This one was a bit easier given our previous look, but again, I really want to stress that **there is no right answer**. All we are attempting to do here is make it easy for you to figure out the naming, but at the end of the day, there's no actual penalty in naming or grouping these however you like. In Phoenix, there is certainly some cost with talking between contexts, but it's not an insurmountable issue.

### Example 3: Google Drive

Google Drive is a cloud file storage provider. You can access files from anywhere in the world on any device, provided you have a Google account. In addition to reading files, you can collaboratively write to files with other users who have permission to read (and write) these files. Further, you can collaborate via tools like comments and sticky notes to provide feedback even if you cannot directly write to the file. Google Drive does not provide universal file capabilities and is mainly focused on reading/writing for documents, spreadsheets, and slideshow presentations.

![Google Drive domain modeling example](/assets/images/google-drive-no-ddd.png)

This one is a bit more complex. We have quite a few more models here with a bit more ambiguity for how we can split things up. Let's start with the easiest one that all exist on the same level: `Document`, `Spreadsheet`, and `Presentation`. How did I group those in the above description? I called them all **files**. Normally we'd red-flag this context because it's too computer-specific of a domain, but remember, this service is _very_ computer focused by its very nature, so it's actually okay that we give this a context like this.

The next layer up we have our `IO` which handles permissions for file read and write access controls and `Comment` which is a special kind of write. Finally, we have the `User` at the very top, which touches everything. Now your gut might tell you that reading, writing, and commenting are **interactions** just like in the Twitter example. But we can't do that for a few reasons:

1. The methods for determining read and write access are encapsulated within `IO` - they are permission grants to unlock certain functionality, but `IO` in of itself is not an interaction
2. Remember, `User` has access to all of these models. Even though this UML diagram lays everything out in a hierarchy, it's not a simple three-tiered relationship, so in a way, we _interact_ with our files as well by virtue of owning them.

Let's offer a different narrative. We mentioned earlier that we can collaborate with other users to perform actions. The noun here is **collaboration**. Additionally, those actions are governed by **permissions**. Which do we choose?

This is probably the toughest one of all. Collaboration implies an innate connection with others, even though you can manipulate a file without any other users. Whereas permission makes sense with both user interaction and `IO` but seems to leave `Comment` out in the cold. The key clue we have is that **`Comment` is a subclass of `IO`, so we can lower the priority of making `Comment` work for our context**. In other words, since we know permission is a sensible context for both `User` and `IO`, it stands to reason that **permission** will be a good context name because `Comment` is a type of `IO` in this domain.

![Google Drive domain modeling example with contexts](/assets/images/google-drive-ddd.png)

This provides lots of ability to expand with both file types and ways of handling things centered around the `User`, such as authorization and authentication. Both contexts also pretty cleanly separate the levels of hierarchy between data models.

## Wrapping up

Let's summarize our heuristic into three simple steps:

1. **Map the domain.** We used UML diagrams to lay out the hierarchy of our models to provide visual clues for possible contexts.
2. **Horizontal or vertical alignment provides clues.** If something falls along the same plane or area of your diagram, you can bet it's a context.
3. **Describe that alignment in one word, preferring the noun with the more specific domain.** This is the hardest part, but the idea here is to take the description of our application and pick out the nouns (and sometimes verbs) that associate our related models. From there, we simply prune the word that has the clearest message with the least ambiguity.

Naming contexts is hard. It can be made easier if you follow these guidelines the next time you write an application using domain-driven design.

Make sure you don't stop with just your first iteration. Growing applications require consistent refinement. Sometimes you'll need to split off a context into multiple contexts, while other times you'll need to consolidate contexts. Don't be afraid to go in either direction. Finally, don't be afraid to be wrong, because there is no definitive right answer either. Good luck and happy naming!
]]></content:encoded>
      <dc:date>2018-05-30T23:25:00+00:00</dc:date>
    </item>
    <item>
      <title>How to Load Images to Cut Page Time</title>
      <link>https://www.adamconrad.dev/blog/how-to-load-images-to-cut-page-time/</link>
      <description><![CDATA[There are lots of ways to improve your site’s performance. Serving your static assets from a CDN, using GZip or Brotli compression, and concatenating your CSS/JS files should be on the top of your list. One thing that may not be on your radar is lazy loading your images.

]]></description>
      <pubDate>Thu, 24 May 2018 18:25:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/how-to-load-images-to-cut-page-time/</guid>
      <content:encoded><![CDATA[There are lots of ways to improve your site’s performance. Serving your static assets from a CDN, using GZip or Brotli compression, and concatenating your CSS/JS files should be on the top of your list. One thing that may not be on your radar is lazy loading your images.

## What is lazy loading?

Lazy loading images involves programmatically inserting images into the DOM only when the image becomes visible within the viewport. It is “lazy” because we choose not to download the image data until it is viewed. This can be extremely beneficial for pages like blogs or landing screens, where only a fraction of your audience will view the entire content.

**How beneficial is lazy loading?** We created a before and after page to illustrate how important it can be to lazy load images. Our example is a sample blog post. Blogs are a great example because they often have lots of images, and most of them are below the fold (outside of the viewport).

[Example blog post (before lazy loading)](https://adamconrad.dev/test-lazyload-slow/)

[Example blog post (after lazy loading)](https://adamconrad.dev/test-lazyload-fast/)

Did you notice the difference? If you’re not convinced, just take a look how much a difference our lazy loading script impacted our site’s performance:

![](https://cdn-images-1.medium.com/max/3998/1*Nt8TzUj5bFOf1m70-5-zgQ.png)

With just one tiny script, we went from an F to an A on MachMetric’s performance grade. We reduced our page load time by **over 90%**. Our page size reduced by **over 95%**!

![](https://cdn-images-1.medium.com/max/2000/1*sZHMARGPNP_DQnsrXy8SDg.png)

Ready to start lazy loading images now? Here’s the code, and it’s super simple!

## Lazy loading — the code

Lazy loading is super easy to accomplish with JavaScript. Here’s [all the code](https://raw.githubusercontent.com/deanhume/lazy-observer-load/master/lazy-load.js) you need to start lazy loading today:

    var images = document.querySelectorAll('.js-lazy-image');
    var config = {
      rootMargin: '50px 0px',
      threshold: 0.01
    };
    var imageCount = images.length;
    var observer;

    var fetchImage = function(url) {
      return new Promise(function(resolve, reject) {
        var image = new Image();
        image.src = url;
        image.onload = resolve;
        image.onerror = reject;
      });
    }

    var preloadImage = function(image) {
      var src = image.dataset.src;
      if (!src) {
        return;
      }
      return fetchImage(src).then(function(){
        applyImage(image, src);
      });
    }

    var loadImagesImmediately = function(images) {
      for (image in images) {
        preloadImage(image);
      }
    }

    var onIntersection = function(entries) {
      if (imageCount === 0) {
        observer.disconnect();
      }
      for (entry in entries) {
        if (entry.intersectionRatio > 0) {
          imageCount--;
          observer.unobserve(entry.target);
          preloadImage(entry.target);
        }
      }
    }

    var applyImage = function(img, src) {
      img.classList.add('js-lazy-image--handled');
      img.src = src;
    }

    if (!('IntersectionObserver' in window)) {
      loadImagesImmediately(images);
    } else {
      observer = new IntersectionObserver(onIntersection, config);
      for (image in images) {
        var image = images[i];
        if (image.classList.contains('js-lazy-image--handled')) {
          continue;
        }
        observer.observe(image);
      }
    }

## How it works

This script works by looking for images that have the class **js-lazy-image**. Those images must have a data attribute called data-src which has the location of the image, just like the normal src attribute would be filled out. The script then counts the number of images with this class, and when the viewport intersects with an observable area that includes the image, the source of the image is loaded from it’s data attributes, and the image renders on the screen! Here’s an example of how you would call lazy loaded images:

    <img alt="Lazy" class="js-lazy-image" data-src="/location/of/my/image.jpg" />

As you can see, this works almost exactly like a regular image, but you just need to make sure to add on the special JavaScript class and add a **data-** prefix in front of your src attribute. And that’s all it takes to go from this:

![](https://cdn-images-1.medium.com/max/3998/1*ZIW_GhTLsCzE23pXO_7ulg.png)

To this:

![](https://cdn-images-1.medium.com/max/3998/1*jVANxQdyxpl8UQ3gmmLr8Q.png)

## How the script works — section by section

Still curious as to how the script works? Let’s break it down by section:

    var images = document.querySelectorAll('.js-lazy-image');
    var config = {
      rootMargin: '50px 0px',
      threshold: 0.01
    };
    var imageCount = images.length;
    var observer;

Here we initialize our variables. Remember that class we added to our images? This is where we collect our images. The configuration will be used later with our observer class (don’t worry if you don’t know what that is, we’ll get to that in a second). Finally, we store the image count because we’ll be using that a few times throughout our script.

    var fetchImage = function(url) {
      return new Promise(function(resolve, reject) {
        var image = new Image();
        image.src = url;
        image.onload = resolve;
        image.onerror = reject;
      });
    }

Our first function deals with grabbing our images. We make use of [JavaScript Promises](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Using_promises) to asynchronously load our images. If the Promise was resolved successfully, we load the image with the URL for our image source, otherwise we display an error. So where is this called from? Glad you asked…

    var preloadImage = function(image) {
      var src = image.dataset.src;
      if (!src) {
        return;
      }
      return fetchImage(src).then(function(){
        applyImage(image, src);
      });
    }

This image preload function grabs the image source from the data attribute we tacked onto our image tag. If it doesn’t find a source object, no problem, we just stop right there. Otherwise, we fetch our image, and if things go great, we apply the image to the DOM like so:

    var applyImage = function(img, src) {
      img.classList.add('js-lazy-image--handled');
      img.src = src;
    }

Our function simply finds the image we’re about to reveal, adds on a class to let our script know it has been handled, and then adds the URL to the image’s src attribute.

    if (!('IntersectionObserver' in window)) {
      loadImagesImmediately(images);
    } else {
      observer = new IntersectionObserver(onIntersection, config);
      for (image in images) {
        var image = images[i];
        if (image.classList.contains('js-lazy-image--handled')) {
          continue;
        }
        observer.observe(image);
      }
    }

This is the meat-and-potatoes of our script. The core functionality that enables our lazy loading to happen quickly and efficiently revolves around the [Intersection Observer API](https://developer.mozilla.org/en-US/docs/Web/API/Intersection_Observer_API). This API allows us to track changes to the viewport against target elements asynchronously.

Traditionally you would have to do something like this with the [Scroll Event](https://developer.mozilla.org/en-US/docs/Web/Events/scroll) which is both slow and called constantly. To combat this performance hiccup, you might using [debouncing or throttling](https://css-tricks.com/debouncing-throttling-explained-examples/) to limit the number of scroll event requests. But with Intersection Observers, this is all handled for you.

Remember the config variable at the top of our script? This is the configuration for our Intersection Observer. It tells us that when we are within a margin of 50px (vertically) from our image, that is when we want to activate our observation callback. The threshold, as you might guess, is the tolerance for what percentage of the object must be observed in order for the callback to be invoked once the margin is reached. In our case, we chose 1%, which is immediately upon bringing the image tag into view.

So now that we have that background, we can see how this if statement works. If we see that Intersection Observers are a part of the window object, we know we are in a browser that supports this functionality. As of right now, [Intersection Observers are available on all major browser except IE and Safari](https://developer.mozilla.org/en-US/docs/Web/API/Intersection_Observer_API#Browser_compatibility). So if you are on IE or Safari, you will load the images immediately. If you aren’t, we create a new Intersection Observer object with the configuration we provided in the beginning, as well as a callback function to trigger when our target observation is reached.

Finally, we have to tell the Observer exactly what it has to observe for the callback to be initialized. In this case, we are observing all of the images that haven’t already been handled, which are the images that haven’t been applied to the DOM yet (via the applyImages function we saw earlier).

So what does loading images and the observation callback look like?

    var loadImagesImmediately = function(images) {
      for (image in images) {
        preloadImage(image);
      }
    }

For loading images immediately, it’s pretty straightforward. We simply preload all of the images and put them on the screen like we normally would.

    var onIntersection = function(entries) {
      if (imageCount === 0) {
        observer.disconnect();
      }
      for (entry in entries) {
        if (entry.intersectionRatio > 0) {
          imageCount--;
          observer.unobserve(entry.target);
          preloadImage(entry.target);
        }
      }
    }

Our intersection callback is a bit more involved. If we have loaded all of our images that have our lazy loading CSS class, we’re done and we can disconnect from our Observer object. Otherwise, for every observer entry in our IntersectionObserver, we want to activate our images. We do that by ensuring we have reached our threshold. **intersectionRatio** is the property we need to see if our target image element is visible within the threshold we defined in our configuration. If it is, the property returns a 1, otherwise it returns a 0. So if we have landed within our necessary ratio, we have one more image to add to the DOM, which means we can remove 1 from our count of images remaining to load. We can therefore stop observing this image because it’s going to be loaded onto the page. Finally, we use our previously-defined preloadImage to execute our now familiar process of adding the image URL onto the image tag and loading the image into the DOM.

## Next steps

**Lazy loading images is a quick and painless way to drastically improve the performance of your pages that use a lot of imagery.** From there, be sure to [compress your images](https://adamconrad.dev/the-fastest-way-to-increase-your-sites-performance-now/) and use the correct image format so you keep a small footprint. Tools like [MachMetrics](https://www.machmetrics.com/) are a great way to track your performance improvements over time, as well as provide additional suggestions on how to continuously improve the performance of your application.

What other quick wins do you have for speeding up your site? Leave a reply in the comments below!

*Originally published at [www.machmetrics.com](https://www.machmetrics.com/speed-blog/how-to-lazy-loading-images-script-slash-page-load-time/) on May 24, 2018.*
]]></content:encoded>
      <dc:date>2018-05-24T18:25:00+00:00</dc:date>
    </item>
    <item>
      <title>Here is the Full List of My 50+ Remote Job Sites</title>
      <link>https://www.adamconrad.dev/blog/here-is-the-full-list-of-my-50-remote-job-sites/</link>
      <description><![CDATA[Are you looking for a job as a UI developer or UX designer?

]]></description>
      <pubDate>Tue, 15 May 2018 23:25:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/here-is-the-full-list-of-my-50-remote-job-sites/</guid>
      <content:encoded><![CDATA[Are you looking for a job as a UI developer or UX designer?

Do you hate leaving the house?

Do you love leaving the house for places that have sand, surf, or other beautiful features of nature?

Then you should think about getting a remote job.

Remote job positions have become far more common than they once were. It used to be the case that you had to pull some serious strings (or be a complete superstar) in order to swing working from home even part-time. Now there are lots of businesses, large and small, that are embracing the remote workforce.

After starting [my own consulting firm](https://www.anonconsulting.com) I knew I had to expand my reach. After doing an exhaustive search, I found quite a few places to find gigs that are okay with remote workers. In fact, I found over 50 remote job sites that are looking for full-time, part-time, and contract talent.

## Common myths about job sites

Job sites conjure up a lot of mixed emotions. Here's a couple of common myths that people think of when they think job site:

* **Bottom of the barrel jobs** - only the worst jobs get posted on these sites. _Actually: lots of great roles with great salaries are advertised online._
* **So many jobs get lost in the fray** - recruiters are on full blast on job sites. _Actually: yes, but lots of respectable companies post directly on here as well._
* **They are only in major cities** - I live in Oklahoma and everything is in NYC. _Actually: did you read the title? You can work anywhere now!_

Here's an example job I pulled today. Does this look like a crappy job?

![Remote jobs can be awesome](/assets/images/remote-jobs.png)

No matter where you live or what you do, making $200k+ working from home (or the beach) sounds like a dream come true. And I literally spent 30 seconds checking one of the sites I'm going to list below.

Whether you're a UI developer or a data scientist, the future of work is remote and agile, so here is my master list of sites I use to land new contracts:

## The complete list of 50+ remote job sites

* [10x](https://10xmanagement.com/)
* [AngelList](https://angel.co/jobs)
* [Authentic Jobs]( http://www.authenticjobs.com/)
* [Clevertech](http://www.clevertech.biz/)
* [Crew](https://crew.co/)
* [Crossover](https://www.crossover.com/)
* [CXC Global](https://cxcglobal.com/)
* [Dribbble Remote](https://dribbble.com/jobs?location=Anywhere)
* [FlexJobs](https://www.flexjobs.com/jobs/telecommuting-programmer-jobs/)
* [FreelanceDevLeads](https://freelancedevleads.com/)
* [Freelancer](https://www.freelancer.com/)
* [Get Better Luck](https://getbetterluck.com/)
* [Gigster](https://gigster.com/)
* [Github Jobs](https://jobs.github.com/)
* [HappyFunCorp](http://happyfuncorp.com)
* [Hired](https://hired.com/contract-jobs)
* [HackerNews Who is Hiring](http://hnhiring.me/)
* [Indeed](https://www.indeed.com/q-Contract-jobs.html)
* [Interviewing.io (as an interviewer)](https://interviewing.io/faq)
* [Jobscribe](http://jobscribe.com/)
* [Karat](https://jobs.lever.co/karat/d44ab283-c7c0-4bbd-b8c3-4dc0ced64c86)
* [Landing Jobs](https://landing.jobs/)
* [Metova](http://metova.com/)
* [Mirror](http://mirrorplacement.com/)
* [Mokriya](http://mokriya.com/)
* [Moonlight](https://www.moonlightwork.com/)
* [NewtonX](https://www.newtonx.com/)
* [NoDesk](https://nodesk.co)
* [Outsourcely](https://www.outsourcely.com/remote-business-services-jobs)
* [PullRequest](https://www.pullrequest.com/reviewers)
* [Reddit For Hire](https://www.reddit.com/r/forhire)
* [Reddit Jobbit](https://www.reddit.com/r/jobbit)
* [Reddit Web Work](https://www.reddit.com/r/web_work)
* [Remote](https://remote.com/)
* [Remote Base](https://remotebase.io/)
* [Remote OK](https://remoteok.io/)
* [Remote Work Hub](https://remoteworkhub.com/remote-jobs/)
* [Remote.co](https://remote.co/remote-jobs/)
* [Remotely Awesome Jobs](https://www.remotelyawesomejobs.com/)
* [Remotive](https://remotive.io/find-a-job/)
* [Remotus](http://remotus.com/)
* [SkipTheDrive](https://www.skipthedrive.com/)
* [SkillHire](https://www.skillhire.com/)
* [Stack Overflow](https://careers.stackoverflow.com/jobs?allowsremote=True)
* [Toptal](https://www.toptal.com)
* [Upwork (formerly Elance / oDesk)](https://www.upwork.com)
* [We Work Remotely](https://weworkremotely.com/)
* [WFH (Work From Home)](https://www.wfh.io/)
* [WhoIsHiring](https://whoishiring.me/)
* [Work Remotely](https://workremotely.io/)
* [Working Not Working](http://workingnotworking.com)
* [Workstate](http://www.workstate.com/)
* [Working Nomads](https://www.workingnomads.co/jobs)
* [X-Team](https://x-team.com/)
* [YunoJuno](https://www.yunojuno.com/)

That enough places for you? There is so much here that you could apply to 5 jobs on 2 sites every day for a month and _still_ have more work to find!

## Bonus: How to make the most of remote job sites

So now that you have more remote work than you could possibly handle, how do you go about effectively navigating all of these roles and positions?

### Polish up your résumé and have it ready

Having your résumé up-to-date may seem like obvious advice, but there are a few ways to do that update that will help you stand apart from the crowd:

* **Write it in LaTeX** - this is my personal opinion, but I think there's a subtle respect in the software community for documents written in LaTeX. Maybe it's because [Donald Knuth wrote it](https://www-cs-faculty.stanford.edu/~knuth/), or maybe it's because it just looks so slick in PDF form.
* **Get a good, clean theme** - something that is easy to read, clean, and that focuses on just the facts. You can take care of both points by writing a LaTeX resume on [ShareLaTeX](https://www.sharelatex.com/templates/cv-or-resume).
* **Keep it to 1 page, regardless of your experience** - even if you have 25+ years of experience, you can trim it down. No one cares about your associate designer role straight out of college in the 80s.

### Get a professional headshot (for FREE)

Take a look at the bottom of the page. My picture was taken in a professional photo shoot. Did you know you can get a photo just like that for free? I am not kidding you. **Search Craigslist for college photographers looking to build their portfolio**.

![Find headshots on Craigslist for cheap](/assets/images/headshots.png)

Lots of art and photography students will post ads like the one above for dirt cheap (or even free) because it's more important to them to build their portfolio than to charge a premium rate for all of their expensive gear and techniques.

Would a true professional be better? Sure. But not for the price. Remember, this is a photo for your online social networks and job sites (Twitter, LinkedIn, your online portfolio for the sites above), not for your wedding. These sites heavily [compress and shrink your photos](/blog/the-fastest-way-to-increase-your-sites-performance-now/) anyway. All you're looking for is something that will stand out and says you mean business.

### Write a killer proposal/cover letter

Now that you look sharp, you need to make an even sharper introduction. This seems to be the one place where people just completely blow by and ignore altogether. They assume that once their résumé is sent, they can just sit back and forget about the company they applied to.

Instead, you should double down and tell them exactly why you are the perfect fit for the job. What work have you done that required similar skills? Have you worked in a similar industry vertical? Unless this is your first job, you likely have some experience you can draw from to highlight in your cover letter. **Showcase whatever you have that will let the recruiter know that you are the perfect person to fill this role they have.**

### Ace the interview

I'll probably write a follow-up article in the future about the art of the interview. In the meantime, know that interviews are just games; and games can be studied and won. To win, you have to put in the time.

![Use interview books to ace your interview](/assets/images/epi-python.png)

A friend of mine recently got his job at Google by studying one of those interview prep books. It took him about 90 hours of concentrated effort to feel comfortable answering the kinds of problems you see in the toughest interviews. If you put in a month of late night studying and practice problems, you'll be good enough to tackle Google, which means you'll be good enough for any remote job you're looking to land.

## Still no luck? Have them come to you!

If you _really_ want the most woke advice possible, then don't even bother getting in the rat race in the first place.

Wait...what?

Did I just tell you to ignore that massive list of websites I just posted? Well, kind of. At the end of the day, it all comes down to two parties trying to connect to fill a position. You can either go out and find the role, or the role can find you.

How can you possibly get companies to find you?

In addition to a nice résumé and great headshots, you should have a website, complete with a blog and portfolio to showcase to the world. Maybe you've spoken at a few local meetups - are those videos online? Do you have a podcast? This is a topic worth an entire series of articles, but the basic idea is that you want to be a known quantity online. **If you put in a lot of effort to get your name out there, eventually people _will_ find you.**

In the meantime, start applying today with the sites above! Did I miss any? Which ones are your favorite?
]]></content:encoded>
      <dc:date>2018-05-15T23:25:00+00:00</dc:date>
    </item>
    <item>
      <title>The Fastest Way to Increase Your Site's Performance Now</title>
      <link>https://www.adamconrad.dev/blog/the-fastest-way-to-increase-your-sites-performance-now/</link>
      <description><![CDATA[The web is getting bigger. It is also getting slower. The funny thing is, every single day I’m visiting new sites to try out a new product or read a blog, and I find the same performance issues over and over again. All I have to do is check out DevTools to see that most images are way too large.

]]></description>
      <pubDate>Wed, 25 Apr 2018 04:25:00 +0000</pubDate>
      <guid>https://www.adamconrad.dev/blog/the-fastest-way-to-increase-your-sites-performance-now/</guid>
      <content:encoded><![CDATA[The web is getting bigger. It is also getting slower. The funny thing is, every single day I'm visiting new sites to try out a new product or read a blog, and I find the same performance issues over and over again. All I have to do is check out DevTools to see that most images are way too large.

What baffles me is how easy it is to fix this. _The_ fastest way to increase the performance of any site is to **losslessly compress images** that are being displayed. In fact, it's so easy, anyone on your team, regardless of coding experience, can make a dent in improving performance.

## Compressing images should be automatic

The [average page size](https://www.machmetrics.com/speed-blog/average-page-load-times-websites-2018/) is 1.88 MB, with the majority of that weight coming from images. So if the bulk of a site is images, we should be focusing on image optimization before we focus on things like database indexing, caching, or other more advanced forms of performance optimization.

[![Performance of top 100 e-commerce sites via Pingdom](/assets/images/top-100-performance.png)](https://royal.pingdom.com/2018/03/07/web-performance-top-100-e-commerce-sites-in-2018/)

## The 4 levels of image optimization

There are four major levels of effort required to reduce the size of image data. Each level builds off of the previous one, so be sure to complete level 1 before moving onto level 2.

The first level requires the least amount of work (which can be accomplished in minutes). The fourth level requires all of the previous steps as well as its own set of tasks. This can take hours, if not days, depending on how image-heavy your content is.

Each level provides diminishing returns. Simply completing one level will do more for your performance than the next level. Which is why it's important not to skip any steps.

### Level 1: Lossless compression

The first thing you should do is grab a copy of [ImageOptim](https://imageoptim.com/). This tiny application is your new best friend. ImageOptim will scan every image within a folder you select, and automatically compress the image, removing all of the useless data.

In the context of this blog, useless data refers to anything that doesn't contribute to the visual quality of the image being displayed. In fact, the term lossless refers to [the ability to reconstruct the original data from the compressed data](https://en.wikipedia.org/wiki/Lossless_compression) without losing any information needed to render that data.

For example, did you know that a photo can store the name of the author of that picture? That comment might be useful in protecting the copyright of the photographer, but it has zero effect on what your application will look like to the world. That metadata is **lossless**, and if you were to run lossless compression on that JPG photograph, it would strip out this and [dozens](https://www.photometadata.org/META-Resources-Field-Guide-to-Metadata) of other metadata fields.

What you may not realize is that this kind of metadata makes up a very large part of the image's total size. This is exactly why lossless compression is so important. **Removing lossless data does not affect the visual quality of your images, but it can drastically reduce their size**. And the smaller the image, the less your browser has to download to retrieve from the server where the image is hosted.

That is why running ImageOptim should be a no-brainer. It only takes a few minutes to run and provides potentially big savings in size without a reduction in quality.

### Level 2: Targeted lossy compression

The next level requires a little more work, but even more savings. At this point, you'll have to make specific fixes for each image type. We'll break this down for the 3 major image types (and if you aren't using one of these three, make sure you check out the next step):

![Toggle metadata in ImageOptim preferences](/assets/images/image-optim-metadata.png)

#### JPG

If you've already run a JPG through ImageOptim, you'll have to download another piece of software. There are additional options for JPGs in ImageOptim preferences, such as removing metadata, but I've found that removing color profile metadata actually colorizes images to include a bit more greens and yellows.

So which should you choose? [JpegMini](http://www.jpegmini.com/app) handles both lossless and lossy compression and allows you to resize your images.

With regards to image sizing, **use the correct image size**. If you have an image that is 512x512 pixels, but the image is only being used in a 64x64 canvas, scale down that image.  Worried about Retina displays? Check out level 4 for more details.

**Another tip: Reduce image quality for JPGs down to 80% by default**. JPG is a lossy format in general, but a very [in-depth analysis of JPG compression quality](http://regex.info/blog/lightroom-goodies/jpeg-quality) shows that there is effectively zero difference in visual quality, but _significant savings in file size_.

#### PNG

Before you need to download anything, you should strip PNG metadata in ImageOptim. This is safe to do for PNGs and I've yet to see a PNG image that looks off by removing metadata.

[OptiPNG](http://optipng.sourceforge.net/) is the next level PNG optimizer after ImageOptim. It has far more algorithms for compressing PNGs and tries far more options to get a truly optimal PNG file size.

![PNG color table](/assets/images/color-table.png)

In addition to compression, PNGs are far more sensitive to **using the correct color profiles**. If you have a black-and-white PNG, using a colorless profile will save a ton of space. From there, **choose the correct color palette size** to restrict the number of colors to an acceptable range. Applications like Photoshop will show you a preview when saving your image, so feel free to play with color palette sizes until you see something that reduces the number of colors without reducing the image quality.

#### GIF

Again, ImageOptim will do a solid job optimizing all image types, including GIFs. However, for a more customized compression technique, take a look at [FileOptimizer](https://nikkhokkho.sourceforge.io/static.php?page=FileOptimizer) with the additional plugins for [GifSicle](http://www.lcdf.org/gifsicle/) and [GifSicle Lossy](https://kornel.ski/lossygif ). These combined seem to do a good job of maintaining image quality while drastically reducing the size of the file.

### Level 3: Convert to the correct image type

<img src="/assets/images/jpegxr-logo.svg" alt="JPEG XR logo" width="100">
<img src="/assets/images/jpeg2000-logo.svg" alt="JPEG 2000 logo" width="100">
<img src="/assets/images/webplogo.png" alt="WebP logo" width="100">

At this point, you've already exhausted even the most advanced compression optimization techniques. And while these programs do all of the work for you in a matter of seconds, the truth is you may need to scrap your images altogether if they are in the wrong image format.

How do you know if they're in the right format? Follow this simple questionnaire and then save your image into the new file format (and then repeat levels 1 and 2 to compress your new images!).

#### Does it look like a photograph? Use a JPG

JPGs were meant to be used with photographs. If you have avatars of your team, photographs of your office, or other real-life images, make sure they are in the JPG format.

#### Does it look like a computer-generated graphic or drawing? Use a PNG

Everything else can be a PNG. Other formats may provide better quality, but if you're serving a web or image application, a PNG will do and is universally read by every kind of application. The one exception to this would be icons...

#### Does it look like an icon? Use an SVG

Icons are also computer-generated, but the difference is that icons are generally used alongside typography. If you think the image will be used in a menu, on a button, or is meant to symbolize something, it's probably an icon, and icons will benefit from being SVGs because they can be scaled along with your type and lose 0 fidelity. They will look just as crisp as your fonts and will be much smaller as SVGs as well.

#### Are you supporting the latest browsers and don't care about Firefox? Use WebP, JPEG 2000, and JPEG XR

Finally, there is a push for next-generation image formats. JPG and PNG have been around for more than two decades, and it's about time we have some new formats that innovate to maintain image quality without bloating our applications. Even better, they don't require you decide between image type. For example, WebP works great for both photographs and computer-generated images.

The downside is that support is fragmented across devices and browsers. [WebP](https://developers.google.com/speed/webp/) was made by Google, so it's naturally designed only for Chrome and Chrome mobile. JPG also has evolved formats, but [JPEG2000](https://jpeg.org/jpeg2000/index.html) is only supported by Apple (Safari and Safari Mobile), while [JPEG XR](https://jpeg.org/jpegxr/index.html) is only supported by Microsoft (IE and IE Edge).

What about Firefox? There is no next-gen format for this browser, but they do have a [2-year-old bug ticket](https://bugzilla.mozilla.org/show_bug.cgi?id=1294490) to implement WebP and it is assigned, but who knows when this will land.

### Level 4: Source sets & fallbacks

If you've chosen the correct image and you've compressed the heck out of it, you'll want to make sure it's being served on the right device in the right aspect ratio. I alluded to this back in Level 2, but if you have concerns about Retina displays like MacBook Pros, or even x3 quality for the newest iPhones, you'll want to ake multiple copies of your image in all of these formats.

Going back to the previous example, if you serve avatars in a 64x64 JPG format, you'll also want to make copies of dimension 128x128 for Retina _and_ 192x192 for Retina x3. The best way to do this is to **start with a larger image and scale down, rather than scale up**. You know those crime dramas where they ask the hacker to "_Zoom! Enhance!_"? We all know that doesn't work in real life, and that same thing holds true for your images - you can't add clarity where there was none in the first place.

Instead, start with the original source image (say, 512x512) and scale down to 192, save a copy, then 128, save another copy, then 64, and save that last copy. This will result in a less blurry, albeit still lossy (because you are removing information in the form of pixel fidelity) set of images.

With all of these duplicate, scaled images, how do we tie this all together? The image attribute known as `srcset` comes to the rescue:

```
<img src="avatar.jpg" srcset="avatar-sm.jpg 1x, avatar-md.jpg 2x, avatar-lg.jpg 3x" alt="Your responsive photo avatar">
```

`Srcset` is pretty amazing. It will always default to the original `1x` magnifier like a normal image tag would. However, if it does find the other images, it will apply them to the given aspect ratios for your device. In other words, if you are viewing the photo on a 2015 MacBook Pro, the browser will select `avatar-md.jpg`, but if you are on an iPhone X, it will select `avatar-lg.jpg`. And if you're in a military bunker using IE8, it will fall back to `avatar-sm.jpg`.

`Sizes` is another property for responsive images but it relies on the width of the device rather than the pixel density. The format is the same:

```
<img src="bg-slim.png"  srcset="bg-slim.png 320w, bg-wide.png 900w" sizes="(max-width: 600px) 320px, 50vw" alt="Your responsive background">
```

You specify an image, space, the size the image should be displayed, with a `w` descriptor for the `srcset`, and then, using the `sizes` attribute, specify the media queries at which point the various sources should be used.

The only downside? **`Srcset` is not supported in IE**. It _is_ supported in IE Edge, and the `sizes` attribute is supported everywhere. In my honest opinion, this isn't something to worry about because all of the Retina devices using IE are already new enough to support IE Edge. Anything that still requires IE11 and down likely doesn't have a Retina display anyway (unless it is connected to an hi-density external monitor) so you likely won't run into this problem being a real blocker for you.

## Something is better than nothing

This is not an exhaustive list of image optimization techniques nor is it meant to be a prescriptive formula. **Even if you only run ImageOptim on all of your images in your application, you could be saving upwards of 60-80% of what your users have to download.**

Images, audio, and video, _not_ your source code, will comprise the largest space for your application by far. Effectively choosing, compressing, and displaying your images will have a marked impact on both application size and performance for your end user experience. The best part is it only takes a few downloads and a few more seconds to run your images through a very easy set of tools that will provide an instant improvement without sacrificing quality.
]]></content:encoded>
      <dc:date>2018-04-25T04:25:00+00:00</dc:date>
    </item>
    <dc:date>2022-03-29T14:01:00+00:00</dc:date>
  </channel>
</rss>