John’s Blog

Very nice response to Marc Andreesson’s post last week by Ezra Klein at Vox:

So let me end with my answer to Andreessen’s question: What should we build? We should build institutions biased toward action and ambition, rather than inaction and incrementalism.

But that means doing the difficult work of reforming existing institutions that aren’t going anywhere. You can’t sidestep the existence of the government, as too many in Silicon Valley want to do. You have to engage with it. You have to muster the political power to rebuild parts of it. And then you need to use the government to make markets competitive again.

But legislators on both sides prefer the status quo because it gives them power when they’re in the minority, and because they’re more afraid of what their opponents might do than committed to what they’ve promised to do. The allure of what they could build isn’t as powerful as the fear of what the other side may build.

My new iPad Magic Keyboard arrived yesterday. I’ve been waiting for this thing patiently since it was announced last month. Along with the Airpods, this is one of the Apple products I’ve been most excited about in the past few years.

Some initial quick thoughts follow…

It feels really nice, so much better than the squishy keys of the old keyboard cover. It feels like I’m typing on a real computer now, instead of some bolted-on after thought accessory.

I work in the mornings before the kids wake up often, usually on the iPad in the kitchen. It’s so nice to have the keys lit up so I can actually type in the dark and see what’s going on. The backlit keyboard is a wonderful addition.

The viewing angle is slightly more adjustable than the keyboard cover, but not by very much for my practical use. I guess it’s nice to be able to fold it into itself a bit more, but I’m not sure I’ll ever use that, especially when just using this as a laptop replacement and typing on a table top. I actually wish that it opened a bit more for typing while standing up at a counter-height surface, but that’s a minor quibble.

These keys sure do feel nice! The mechanism under the key is crisp and responds very well.

It’s going to take a bit to get used to the iPad screen hanging out right above the number keys. It seems I lift my fingers up when typing more than I would have predicted so I’m hitting the bottom of the iPad screen more than I’d care to. I think with some practice this will go away.

Initially when I attached this keyboard to my 2018 iPad, it did nothing. Oh shoot, did I get a dud? I was so excited to play with this thing. No, it works fine. It turns out that I needed to run a software update on my iPad. After a slow update and reboot, I’m in business. Halfway through the reboot the keyboard just lit up… ahh.

I still don’t understand why the need for a hardware ‘globe’ button that enables the emoji keyboard. The iPad keyboard cover I was using prior has it as well. Why? It’s in the same place as the Fn key on a standard Mac laptop, so I instinctively hit it when I’m doing text manipulation. (I never realized how often I use Fn-Delete to remove characters in front of the cursor. It turns out that Control-D does the same thing, so I need to switch my muscle memory to that instead.) Lucky for me, I can use the keyboard settings in iOS to disable the silly globe key so it has no effect when pushed. Waste of space, but I can handle that.

The trackpad… wow there’s a trackpad for iOS! This is so delightful. It’s so natural for me to reach for the trackpad now instead of having to lift up my fingers and drag the screen around when I’m typing. It sounds silly, but it makes for such a nicer experience.

This week I was deploying a routine update to Heroku, but the app wouldn’t deploy. Here is the message that was displayed in the build log:

-----> Discovering process types
       Procfile declares types     -> release, sidekiq, web
       Default types for buildpack -> console, rake
-----> Compressing...
 !     Compiled slug size: 502.5M is too large (max is 500M).
 !     See: http://devcenter.heroku.com/articles/slug-size
 !     Push failed

Slug size is too large was the culprit. 502 MB! The app is decently sized but that seems way off to me. According to rails stats the app is about 58,243 lines of code. Not small, but certainly not huge. And these are mostly source code plain text files too so file size should not be an issue at this scale.

Investigating the Sizing

I used a basic du -sh * command to check the basic file sizes of the folders in the app’s repository to see what’s going on. Here’s the truncated output:

12M    app
352K    config
1.3M    db
376K    lib
332K    public
260M    node_modules
 78M    tmp
226M    vendor

So my app is relatively small, only about 17 MB. The rest of the file size comes from the various dependencies. By some standards, I don’t really even think this app has very many dependencies. There are a few dozen Ruby gems in the Gemfile and there’s nothing out of the ordinary here. Rails, Postgres, Nokogiri, etc. All things you would expect. (Although side note, the google-api-client gem is 61 MB on its own. Yikes, by far the largest gem in there. That’s something to work on for another day.) All of the gems are in vendor/ so the file size there checks out as reasonable.

The tmp directory is pretty big here too, but that’s ok. It is mostly filled with bootsnap caching files to make the app load quicker too. This would be an easy 78MB to clear out if it was the culprit, since I don’t really care how long the app takes to boot, but let’s reserve clearing it for later.

Even that massive node_modules directory isn’t really problem, since Heroku slugs are compressed before this slug size is calculated. Those dependencies compressed down are only about 80MB.

So there’s something up here. Even with my most conservative calculation the slug size should be max of around 300MB after compression. Which is still too big in my opinion for an app of this size, but still nowhere near the limit of 500 MB.

There is a nice Heroku plugin that allows you to download slugs from your builds. It’s a very handy way to download the exact slug from Heroku’s filestore on S3. I downloaded the slug in question to give it a look.

Using the same du command within the extracted slug files, there was one major diference:

961M    public

Whoa. 961MB in the downloaded and unzipped slug, but only 332K in my project’s repo itself. It turns out that inside the public/assets directory there were hundreds of asset build files dating back years. The app itself is a few years old and it looks like Heroku had stored the compiled assets in the slug for each release since the app was created. Yikes.

This is a known feature on Heroku and it is called the build cache. I was mistakenly under the impression that each app deploy would cause a new slug to be built, including the asset compilation. The build cache feature makes sense: why waste time re-downloading files and other dependencies on each build when they can be cached? But caching the output of asset creation until the end of time? That seems excessive. So let’s clean it up.

The Fix

It turns out there’s an easy fix when build cache is your problem. There’s another Heroku plugin for managing the repo that contains your app’s code. Within this plugin is the purge-cache utility, which clears the build cache.

Here’s how it works:

heroku plugins:install heroku-repo
heroku repo:purge_cache -a appname

I redeployed the app and that shaved off a few hundred MB from the slug size. We’re back in business here. But I still couldn’t get past that massive node_modules/ directory. That was 260MB of modules only used for compiling assets. Once the assets were compiled, those modules are no longer used. (This particular app does not use any node_modules at run time, they are all precompiled with webpack.)

There is a community-developed buildpack for Heroku called post-build-clean that gives us the ability to remove files from the slug after the build command finishes running. This is slightly different than the standard .slugignore file which won’t even allow you to use those files during the build phase. I installed the buildpack, added node_modules to the .slug-post-clean file and tried to deploy again.

This time around, it looks much more reasonable to me:

-----> POST BUILD CLEAN app detected
Removing directory node_modules/ from slug
-----> Discovering process types
       Procfile declares types     -> release, sidekiq, web
       Default types for buildpack -> console, rake
-----> Compressing...
       Done: 87.4M
-----> Launching...
 !     Release command declared: this new release will not be available until the command succeeds.
       Released v1019

For a medium-sized app 87 MB seems like a good size to me. The asset files continue to build up after the compilations step, but it’s going to be a while until I need to clear the cache again.

Marc Andreessen:

Every Western institution was unprepared for the coronavirus pandemic, despite many prior warnings. This monumental failure of institutional effectiveness will reverberate for the rest of the decade, but it’s not too early to ask why, and what we need to do about it.

Many of us would like to pin the cause on one political party or another, on one government or another. But the harsh reality is that it all failed — no Western country, or state, or city was prepared — and despite hard work and often extraordinary sacrifice by many people within these institutions. So the problem runs deeper than your favorite political opponent or your home nation.

Part of the problem is clearly foresight, a failure of imagination. But the other part of the problem is what we didn’t do in advance, and what we’re failing to do now. And that is a failure of action, and specifically our widespread inability to build. […]

Building isn’t easy, or we’d already be doing all this. We need to demand more of our political leaders, of our CEOs, our entrepreneurs, our investors. We need to demand more of our culture, of our society. And we need to demand more from one another. We’re all necessary, and we can all contribute, to building.

Every step of the way, to everyone around us, we should be asking the question, what are you building? What are you building directly, or helping other people to build, or teaching other people to build, or taking care of people who are building? If the work you’re doing isn’t either leading to something being built or taking care of people directly, we’ve failed you, and we need to get you into a position, an occupation, a career where you can contribute to building. There are always outstanding people in even the most broken systems — we need to get all the talent we can on the biggest problems we have, and on building the answers to those problems.

Interesting analysis of the sports world, specifically the in-stadium experience, in a post Covid-19 world by Alex Speier in the Boston Globe:

To maintain 6 feet on all sides, you'd likely need multiple empty seats and multiple rows between fans — some of which could potentially be offset by having, say, a family of four from one household sitting together in a block. Crowd composition could be altered further by profiling. Might teams discourage those at greater risk of serious complications from COVID-19 — those over 60, or those with preexisting conditions — from going to games? Would teams restrict tickets to — or feature different seating plans for — those who could document they had developed antibodies to the coronavirus?

Decontamination of stands will have to become a staple of stadium operations. Hand sanitizer will become omnipresent in concourses. Cleaning staffs would have to be vigilant about the "high-touch" areas of facilities — including railings (both in stands and on escalators) and elevator buttons.  Might there be a requirement for spectators to wear masks? If so, masks with team logos might replace caps or jerseys as the most frequently seen form of team apparel.

It's a quintessential part of the stadium experience: A hot dog passed from vendor to fan to fan to fan, with cash flowing back in the other direction. In all likelihood, that familiar ritual will be gone. "They'll have to have no stadium vendors," said Zimbalist. "They're not going to have people passing hot dogs down or passing anything down. That has to stop."

via Peter King's Football Morning in America

Some extraordinary photos of New York City this month by photographer Scott Medway on Exposure.

For the past few weeks, I've been helping out a friend with some technical recruiting. This is a bit of new territory for me and with that always comes new things to learn. I've done my fair share of hiring in the past, so it isn't completely foreign but this situation is unique for me.

The process is exhausting and difficult but in the end you're bringing a new person onto your team to help your company grow. So it's worth it. You keep pushing through because that one next person can really help your company grow.

But I've never tried to hire someone for someone else's team. It feels counterintuitive in a way. But I'm wrapping my head around it so far. I'm helping talented people find a great job and that's wonderful. I'm also helping someone else build a team to do important work. I'll only be involved from afar, but the end result is the same.

The Industry

The technical recruiting world is broken and awful. The vast majority of people that I've encountered doing this job are playing a strict numbers game. They blast out job postings, candidates, and poorly researched e-mail introductions by the thousands just hoping for a few suckers to respond.

From a candidate's side it's a beat down. I've long ago disabled all notifications from LinkedIn and I have a number of pattern matching filters on my e-mail addresses to keep this stuff from ever reaching my inbox. I'm not trying to be rude, but there is so much garbage out there that it's hard to find a signal in that noise. Every few weeks I'll go into LinkedIn and my message queue is filled with recruitment notes for "once in a lifetime" and "top tier" positions at great "brands". If these positions were that unique, I wouldn't get 20 of them a week.

From the company's side, it's also a beat down. Many of the companies are actually doing interesting work. They have a real problem on their hands and they have a business opportunity they could accomplish if they could just find a few good people to focus on it. Many (but certainly not all) of the hiring managers in these companies are not technical and feel unequipped to find the right people for the positions. To solve the problem there are usually both internal company recruiters and external staffing and recruiting firms. Both have their benefits and both approaches can be completely terrible.

Staffing firms have a less than desirable reputation in the tech world. For good reason too. The numbers game playing out each day is a complete turn off and gives the industry a bad name. It's not usually that they don't care, they just don't know what to do. They are tasked with hiring for a very technical position that they don't understand and don't know how to qualify people for. It's a beat down all around.

Getting Better

I'm not here to solve the world's recruiting problems. I'm passionate about building software and companies around that software. Recruiting is certainly one piece of that puzzle, but it's usually something I leave to others. In short, I'm not leaving my 'day' job to become a recruiter, but I can still help a little bit.

The biggest thing I think I'm contributing to this process is that I know these technical jobs. I've done the jobs myself, I've been in these roles. I've hired for them before in my own companies. I know the tech and I understand the problems the companies are trying to solve by hiring. I also know a thing or two about shipping software and actually accomplishing organizational goals. So I'm using what I know to identify people that I'd hire if I was the company.

The challenging part of finding who I'd hire is: I'm not hiring them! I am just one piece of the process and just one opinion in the room. Sometimes who I'd hire isn't the right fit for a company, and that's totally fine. It's their company and I need to defer to them on what's best. It's hard to present someone I think is great and have them rejected, but that's a part of the deal. I'm working on getting better at this.

The Interview Process

I'm learning about big company interview processes a bit too. The biggest company I've worked for was about 120,000 employees at its peak, and I've slowly moved my career into the opposite direction ever since. I prefer to work in a startup with less than 10 people. So it's a bit different. There's nothing inherently wrong with a big company, as much as it's not for me.

At a bigger company, the interview process is completely different than what I've been exposed to most of my career. For the past decade or so I've almost exclusively been hiring for companies that I own or have built. This is a very different process than hiring for a company that's been around for decades and is publicly traded, for example. I would obsess on every hire and worry about how they would affect the company's vibe and culture. I cared more about their personality and past experience than how technical they were. I can teach programming, but I can't teach someone be kind to their coworkers if they don't think that's worthwhile. With a bigger company, culture is still important, but it's very different than a startup. Hard skills seem to matter more than the soft ones.

My interview style is to speak conversationally with the person and just try and get to know them. I want to know how they think. How do they solve problems? What problems have they solved before? What did they learn from those problems? Sure, we can get technical here. If they built a cool API I'll most certainly ask how they built it. What did they build it with? How did it scale? How did you decide to use this piece of tech over this other one? There aren't right or wrong answers here, for the most part. I'm imagining this person in the role I'm hiring for and trying to see how their past experiences and best practices align with mine and those of my team.

I'm not into tricks and riddles and code algorithm puzzles. Not one bit. I don't care if you know some specific computer science term or how to best write some algorithm. Most good developers I know spend a lot of time Googling and figuring things out. I know I do. If I'm presented with a performance challenge, I'm surely not going to recall some abstract algorithm from college and apply it from memory. I'm going to search for a solution. I'm going to look at forums and read blogs about how others are solving the problem. I'll chat it through with coworkers. Then I'll devise a solution and implement it. Maybe I'll do that whole process two or three times before the best solution presents itself.

So I'd probably be terrible in a coding quiz interview. I might be laughed out of the room if I had to whiteboard some computer science algorithm from memory. I don't need to know that stuff, I just need to know that it's a thing. In the "real world" I'd Google it and solve it within 10 minutes.

But these type of things are quite common in big company land. So I'm learning to adapt. If I find someone I like, it's now part of my job to prepare them for these type of technical interviews. Once I find someone that I think can do they job, I need to also make sure they can pass the interview. These are very different things, but if the end goal is the same, it's a worthwhile effort.

As with everything in this space, I'm not sure if I'm right or wrong. I don't think there is a hard and fast 'right or wrong'. I'm just doing what I know and what's worked for me.

What's Next

As I think I've articulated, recruiting is a big giant mess. It's hard for everyone. There's no perfectly correct way to do this and the only way to get better is to keep trying to get better.

I'm going to keep trying for now. It's a small chunk of my free time, but it's been an interesting ride so far.

Now back to building software.

Jerry Jones (no, not that one) writing about his interview and eventual hiring process at Automattic:

The entire hiring process is text based. Seriously.

Never once did I hear someone’s voice or do a video call. Until my first paycheck arrived, a part of me still believed it was all too good to be true, and it was just an elaborate prank.

They invited me to a slack channel, and I was free to ask questions and talk with the hiring team. They told me how they do what they call “async communication.” You can ask a question, and you may not get an answer for awhile, as the person may be in a totally different part of the world.

I found the idea of a Slack-only interview fascinating. I’m sure it can be difficult waiting for responses in the void sometimes, but this also seems like a great way to also respect the workload of those on the inside doing the interviews as well.

There’s some other interesting parts in the piece about coding tests and trial run too. It’s clear there has been some careful thought put into this process, and that’s nice to see.

From Matthew Panzarino at TechCrunch:

Apple and Google’s engineering teams have banded together to create a decentralized contact tracing tool that will help individuals determine whether they have been exposed to someone with COVID-19.

Contact tracing is a useful tool that helps public health authorities track the spread of the disease and inform the potentially exposed so that they can get tested. It does this by identifying and “following up with” people who have come into contact with a COVID-19-affected person.

The first phase of the project is an API that public health agencies can integrate into their own apps. The next phase is a system-level contact tracing system that will work across iOS and Android devices on an opt-in basis.

The system uses on-board radios on your device to transmit an anonymous ID over short ranges — using Bluetooth beaconing. Servers relay your last 14 days of rotating IDs to other devices, which search for a match. A match is determined based on a threshold of time spent and distance maintained between two devices.

If a match is found with another user that has told the system that they have tested positive, you are notified and can take steps to be tested and to self-quarantine.

This is great news. Any system like this will need deep integration into the operating systems to be effective. Third-party apps can implement some nice features, but the platform itself has to be on board for more widespread adoption. It’s nice to see the two big players in the mobile space cooperating on this level.

They have already started to release some documentation and notes about the APIs. This should be an exciting space to watch over the next few months.

Three weeks down. Who knows how many more to go? We are a bit over three weeks into our family’s social distancing. I’m not sure what to call it really, but we’re staying home and not interacting with anyone outside of the house except for an occasional run to the grocery store. Or to get tequila.

Three weeks ago our schools were still open. Restaurants were still open. There was toilet paper and packs of eggs in stores. If you were sick you could buy Tylenol and Advil. Hospitals had beds available. We were excited to start a new season of tee-ball and I was looking forward to the start of baseball, March Madness, and The Masters.

What a difference that three weeks make. We decided to pull our kids out of school before spring break and they haven’t returned since, and probably won’t finish out the school year. Our county announced a shelter-in-place order requiring everyone to stay home until at least May 20th. March Madness was completely canceled, The Masters is delayed (but probably canceled) and even the Olympics for this year are postponed until next summer. It feels like the whole world is on pause.

For the most part, our time as a family has been great. We’ve never spent this much time all together, and likely never will again. I’ve heard it takes humans a little more than three weeks to develop new habits and routines. This time doesn’t feel new any more and it feels like we’ve established our new routines. We’re slowly figuring out how to do home schooling. We’re adjusting to not being around friends and having things to do outside of our neighborhood. We’re finding new things to cook at home and creative ways to pass the extra time. I’m truly thankful for this time together as a family. As long as the Internet doesn’t go out, we’re doing just fine over here.

I’m anxious about the world around us. I see our family, and others we know, doing everything we can do to prevent the spread of this virus but it’s not enough. There are so many people around us that do not seem to understand what’s going on. They’re playing and interacting like nothing has changed and burying their heads in the sand. I heard one older Texas man say that “we all need to just put on our big boy pants and get on with life.” We continue to have awkward conversations with people about how we can’t meet up for a certain activity now how they can’t come visit right now. I don’t feel like I should have to explain, but I still do. It feels like we’re the ones doing something wrong, and the rest of the world is moving on without us sometimes. But that’s not the case. I know it’s not the case, and soon I hope everyone will understand it as well.

So three weeks down, and on to the next. April is going to be an interesting month. Stay safe out there.

Bill Gates in The Washington Post this week:

There's no question the United States missed the opportunity to get ahead of the novel coronavirus. But the window for making important decisions hasn't closed. The choices we and our leaders make now will have an enormous impact on how soon case numbers start to go down, how long the economy remains shut down and how many Americans will have to bury a loved one because of covid-19.

Through my work with the Gates Foundation, I've spoken with experts and leaders in Washington and across the country. It's become clear to me that we must take three steps.

Very clear and reasonable steps.

It seems about time for the leadership in this country to get its act together and start taking control of this. I’m looking at you too, Texas.

Jahanzeb Sherwani announcing his new venture, Screen:

In 2013, I co-founded a company called Screenhero, which made an app that let pair programmers work together remotely. Through low-latency screen sharing and shared control, it let programmers code together on a Mac or Windows computer. Customers loved us, and voted with their wallets: once we started charging, we reached $1M in annual recurring revenue in 5 months. We had the choice of staying independent, but we opted to join forces with Slack, with the intention of embedding our product inside theirs, to reach more people faster than we could on our own.

In 2015, Slack acquired Screenhero, and I led the team that built Slack Calls: voice, video and screen sharing in Slack. We finally shipped interactive screen sharing almost three years later, but it wasn't as performant as Screenhero, and was eventually removed in 2019. Given that it was used by a tiny fraction of Slack's user-base, and had a high maintenance cost, this was the correct decision for Slack.

The old Screenhero was so great. Slack's video sharing was never the same and I'm so delighted these guys have returned with a new app.

I’m really enjoying this episode of Bill Simmons’ podcast with Eddie Vedder and Jeff Ament of Pearl Jam. Among all of the current crazy in the world it’s nice to hear about something different.

I love the stories of the old days and hearing their perspective on the crisis of the day. Remember when we took for granted that we could all gather in a large crowd and go see a concert together?

Speaking of working remotely from home ... Glenn Fleishman has a new (free) e-book: Take Control of Working from Home Temporarily.

We're in a time of unprecedented uncertainty. In the middle of a global viral outbreak, you were told or asked to work from home—and you've never or rarely had to be productive where you live before. What to do? We're here to take some stress out of your life with a new, free book that details how to set up a home office and balance work and home life for those not accustomed to it.

Some great suggestions on working from home by Matt Stauffer:

Today, I want to talk about remote work—especially right now, as so many people are unexpectedly being told/allowed to work from home—and how so much of it happens in less-than-ideal environments, and what we can do to make the best of it. I'll assume you're working from home, but many of these tips apply in other less-than-ideal remote work environments as well.

I agree with many of his tips. I've been working from home now for over 4 years and it took some getting used to in the beginning. Now I can't imagine anything else.

While working from home takes some getting used to and people may not be as productive as before we have to remember that we're still in the middle of a global crisis, and that doesn't make working easier regardless of your work location.