John’s Blog

Oliver Darcy, writing in this morning's Reliable Sources newsletter:

yet againtweeted

Sara Fischer, reporting for Axios:

The New York Times will no longer use 3rd-party data to target ads come 2021, executives tell Axios, and it is building out a proprietary first-party data platform.

Beginning in July, The Times will begin to offer clients 45 new proprietary first-party audience segments to target ads

This is great news and I hope others follow. It's going to be tough for them to pull away from the giant data providers, but I hope that publishers can do it. Third-party tracking and sharing of user data is gross and a privacy nightmare.

As an aside, I do happen to run a small independent publisher that is partially supported by sponsored ads at Air Mail. When I built the tech for Air Mail, I specifically and intentionally created a system that wouldn't allow any third-party tracking of ad data. We host and serve all of our ads in a first-party and private matter. Tracking clicks and impressions is standard practice for ad servers and ours does it entirely in the background as well. This allows our sponsors to check their numbers without compromising on one ounce of customer data from our readers. I created the type of system that I wouldn't mind using as a reader.

It's not that complicated if you design your ad systems with privacy in mind from the beginning. Here's hoping more of the publishing world catches on.

Ashley Carman for The Verge:

Joe Rogan, comedian and host of one of the most popular podcasts in the world, is taking his show to Spotify. The Joe Rogan Experience will soon become a Spotify exclusive, meaning episodes' full audio and video will only be available through the platform starting later this year. Up until now, Rogan's show has never been available on Spotify, let alone exclusive to any platform.

Spotify is quickly eating up the "podcast" world. Not great. A "podcast" that is only available on one app and does not provide an open feed to access its shows is not a podcast. Maybe we need a new name for these type of things. We need more independent podcast publishers, not consolidation of power into the hands of the few.

Dave Grohl in the Atlantic:

In today’s world of fear and unease and social distancing, it’s hard to imagine sharing experiences like these ever again. I don’t know when it will be safe to return to singing arm in arm at the top of our lungs, hearts racing, bodies moving, souls bursting with life. But I do know that we will do it again, because we have to. It’s not a choice. We’re human. We need moments that reassure us that we are not alone. That we are understood. That we are imperfect. And, most important, that we need each other. I have shared my music, my words, my life with the people who come to our shows. And they have shared their voices with me. Without that audience—that screaming, sweating audience—my songs would only be sound. But together, we are instruments in a sonic cathedral, one that we build together night after night. And one that we will surely build again.

Amen.

The bit about Springsteen in the piece is priceless too.

(via Daring Fireball)

David Peisner, writing about the making of Jason Isbell’s new album for the New York Times:

After all the strife the album caused, it’d be understandable if Shires never wanted to hear it again, but that’s not the case. “It’s the worst recording experience I’ve ever been a part of, but it’s my favorite record he’s made,” she said. “I’d like to say we’re stronger because of it, but we’re not. We just know that our strength is more than we thought.”

Isbell doesn’t think the album was affected by the turmoil he underwent making it but allowed for the possibility he could be wrong. “Maybe you can hear it,” he said. “Maybe the record is better for it. I don’t know. I try not to ask that question because I don’t want to get in a pattern of [expletive] my life up to make better records.”

What a refreshingly honest piece. Tough discussion I’m sure. Isbell is by far my favorite artist of the past 5 years or so. Looking forward to the new album. (Call me old-fashioned, but I don’t listen to singles ahead of the release. I like to wait for the full thing.)

Anil Dash writing about this yesterday’s Facebook SDK issue that crashed many apps:

So, understandably, everybody just plugs in the Facebook code (often called a “library” or more formally, a Software Development Kit, “SDK”) and focuses on the more important features of their app. But while lots of open source code libraries that you might use just perform a certain function in your app, like displaying a picture or formatting some data, this Facebook code also relies on a service on Facebook’s site running properly, too.

Today, that service got broken.

The result of Facebook’s breakdown today is kinda wild: a minor configuration change on a Facebook server that isn’t even visible to regular users made dozens of high-profile apps from some of the biggest companies in the world all start crashing when you open them — even if you weren’t using Facebook at all.

This was supposed to be a great year for live music. Well, every year is a great year for live music. But, for me, any year with a new Pearl Jam tour these days is a good year.

One of my favorite things to do is see my favorite bands and performers live in concert. In a previous life I would travel up and down the east coast to see shows as much as I could. I’ve lost count of the number of times I’ve seen a few bands and still wish I could go to more. It’s harder for me to get out these days (life with small children at home is complicated!) but I still manage to get to a select few shows each year.

And yet, right now I can’t imagine a more dangerous place than a crowd of 500 to 20,000 people sitting together in close proximity. Everyone breathing on one another. Ugh. Gross. This isn’t something we’ve imagined before. Live events are supposed to be fun: a way to escape life and enjoy the music for a few hours. As with everything else right now, it’s a whole new world.

The music industry doesn’t know how to handle what’s happening. None of us do, really, but some are handling it better than others. The music industry is lazy. More specifically, the live music ticketing industry is lazy and has barely innovated in decades.

Why should they innovate? Each year millions of people, just like me, buy tickets as fast as they can when their favorite artists announce new tours. There’s no reason for these providers to change when they are already making money at an alarming rate compared to the actual service they provide. Service charges, convenience fees, and other ticketing add-ons continue to increase in cost, while the service and convenience they offer continue to decline.

As fans, we go online at a specific time to try and fight the bots and scalpers to get decent tickets. The tickets go on sale at 10:00 AM local time on some random weekday when we’re all supposed to be working because the ticket providers still can’t handle a surge of traffic that’s less than Amazon receives in an hour of Cyber Monday sales. And yet their servers still buckle under the pressure. Bots, scalpers, and preferred ticketing vendors get all of the good seats and true fans are left with scraps. Which we always buy because, hey, we’re in the building. And so is Paul McCartney!

Or, if you’re out of touch like me, you don’t even realize when tickets are on sale until that magical time has already passed. This happens to me all of the time. I’m listening to something on Apple Music and I wonder if the artist is touring soon. I check it out and sure enough they are coming soon, but tickets are already mostly sold. Then you’re stuck in the third-party market of StubHub and others; inevitably paying a hefty premium on top of a face-value ticket price to some scalper that never intended to go to the show anyways. There has to be a better way.

Then the coronavirus happens and the live music world is upended. Last night I received an e-mail from Ticketmaster about a show that was supposed to take place back in March:

Hi John,

Unfortunately, the event organizer has had to cancel your event.

The good news is that a refund will be processed automatically for you. Due to the unprecedented volume of cancellations, you should expect to receive your refund in as soon as 30 days.

Please Note: If the tickets were transferred to you, the refund will go to the fan who originally purchased the tickets from Ticketmaster.

Of course the concert has been canceled. It was supposed to take place in March, and it’s now May 2nd. That’s also nice that they are going to refund my purchase. Strangely though, much like unsubscribing from a mailing list, it takes 30 days to refund. But it’s better than nothing.

Except for the last sentence of the e-mail: “the refund will go to the fan who originally purchased the tickets from Ticketmaster.”. Ticketmaster knows that I have the tickets, but it isn’t smart enough to realize that I didn’t purchase the tickets. I bought them from someone else on StubHub (for a premium, sigh) and then received the tickets as a transfer. So according to this it seems that I probably over paid for the concert (which is totally fine) on StubHub, and now the original purchaser is going to receive a refund for whatever they paid. We can only assume that I’ll receive nothing by way of a refund, even though I clearly ‘own’ the tickets in my Ticketmaster account.

The money I’m out for the concert is annoying, but I get that. It’s a minor blip on the coronavirus radar these days. It wasn’t an expensive ticket so I’m not that worried about it.

But the bigger issue is how much of a mess the ticketing industry is. Ticketmaster, building on tech from decades ago, is not prepared to handle simple things like returning an order. StubHub, and others like it, are seemingly still operating on the idea that a ticket is a piece of paper that once transferred to someone retains no historical context of its origin.

The concert industry, like many, is hurting right now. This could be a time of great innovation and overall cleanup. Ticketmaster is a giant. They have immense resources and even more political sway in the industry. What if they took on the task of modernizing the concert ticketing space? That seems unlikely.

The more likely and hopeful scenario is one of disruption from a new player in the market. Cloud services are ubiquitous. Mobile ticketing solutions are available to everyone with a smartphone.  The databases and integrations required for this type of innovation are not complicated from a systems design perspective.

The live music industry is at a standstill right now. It’s going to be months or years before concerts are back to the way we left them. That’s plenty of time to start innovating and clean up this mess.

As a fan, I hope someone is up to the challenge. Here’s to 2021 being the best year for live music and ticketing yet.

David Heinemeier Hansson joins the discussion on evaluating programmers with a skills test. At Basecamp, they prefer a late stage take-home challenge that mirrors the actual work that will be performed in the job if hired. As usual from Basecamp, this is very well thought through and sounds like a nice process for the company and the candidate alike:

There’s no perfect process for hiring great programmers, but there are plenty of terrible ways to screw it up. We’ve rejected the industry stables of grilling candidates in front of a whiteboard or needling them with brain teasers since the start at Basecamp. But you’re not getting around showing real code when applying for a job here.

So we whittle the group of candidates down aggressively first. This means judging their cover letter and, to a far lesser extent, their resume. For the opening we had on the Research & Fidelity team, we gave 40 people the take-home test, and even that proved to be too many. For the opening we had on the Security, Infrastructure & Performance team, we only gave 13 people the take-home test. That felt better. In the future, we’ll target fewer than 20 for sure.

Then there’s the assessment itself. I’ve heard many fair complaints that companies are asking candidates to complete massive projects that may take 20-30-40 hours of work, which is all unpaid, and which might be difficult for candidates to fit in with their existing job and life. Yeah, don’t do that. Asking someone for forty hours of work product, without pay, which might well go nowhere, is not what we do or advocate at Basecamp.

Brent Simmons, who has been building and shipping exceptional software for decades, has been writing about his recent reentry into the job market and practicing for coding skill tests during interviews. I’ve followed Brent’s work for as long as I can remember and his blog is one of my favorites. A few days ago, he described the pain of studying for code interviews:

I don’t have a CS degree, but I have decades of experience — I know what a linked list is, for instance, and could write one by hand easily if called to. In a few different languages, even. I could talk about the trade-offs between a linked list and a contiguous array. Etc. I’ve got all that.

My style of coding is to break problems into steps and make it super-obvious to other people — and future-me — what the code is doing. I like to write code so clear that comments aren’t needed.

He then describes how he approaches problems normally during the course of his work and how that is very different from what these technical interviews are looking for. When I wrote last week about technical recruiting I was thinking about this same problem. This style of technical interview is set up to quiz people on very low-level puzzles and graded on a scale designed for a computer science exam, rather than actual day-to-day aptitude for real job.

Brent followed up yesterday with another post:

There’s a whole small industry to help people prepare for these tests — so it’s not like you’re getting the authentic programmer showing up. You’re getting the person who’s prepared for one of these.

Because of that, an interviewer is even less likely to learn how a candidate approaches solving a problem. Instead, they’ll learn how well the candidate prepared to make a good impression — which tells you nothing about how they’d actually solve a problem.

Daniel Jalkut, on Twitter, reacting to Brent’s posts:

No matter how experienced we are, the prospect of code-related interview questions provoke fear in us. I don’t know if any other industry is like this, where inrterviews are designed as “gotcha” traps, designed to make fools out of geniuses.

I understand the need to see how someone codes before hiring them to join your company. But it shouldn’t be tricky. And it should focus on the actual job itself, which for most of software development, is just solving hard problems. I’d much rather hear about how someone solved a problem than if they know a specific sort algorithm.

Very nice response to Marc Andreesson’s post last week by Ezra Klein at Vox:

So let me end with my answer to Andreessen’s question: What should we build? We should build institutions biased toward action and ambition, rather than inaction and incrementalism.

But that means doing the difficult work of reforming existing institutions that aren’t going anywhere. You can’t sidestep the existence of the government, as too many in Silicon Valley want to do. You have to engage with it. You have to muster the political power to rebuild parts of it. And then you need to use the government to make markets competitive again.

But legislators on both sides prefer the status quo because it gives them power when they’re in the minority, and because they’re more afraid of what their opponents might do than committed to what they’ve promised to do. The allure of what they could build isn’t as powerful as the fear of what the other side may build.

My new iPad Magic Keyboard arrived yesterday. I’ve been waiting for this thing patiently since it was announced last month. Along with the Airpods, this is one of the Apple products I’ve been most excited about in the past few years.

Some initial quick thoughts follow…

It feels really nice, so much better than the squishy keys of the old keyboard cover. It feels like I’m typing on a real computer now, instead of some bolted-on after thought accessory.

I work in the mornings before the kids wake up often, usually on the iPad in the kitchen. It’s so nice to have the keys lit up so I can actually type in the dark and see what’s going on. The backlit keyboard is a wonderful addition.

The viewing angle is slightly more adjustable than the keyboard cover, but not by very much for my practical use. I guess it’s nice to be able to fold it into itself a bit more, but I’m not sure I’ll ever use that, especially when just using this as a laptop replacement and typing on a table top. I actually wish that it opened a bit more for typing while standing up at a counter-height surface, but that’s a minor quibble.

These keys sure do feel nice! The mechanism under the key is crisp and responds very well.

It’s going to take a bit to get used to the iPad screen hanging out right above the number keys. It seems I lift my fingers up when typing more than I would have predicted so I’m hitting the bottom of the iPad screen more than I’d care to. I think with some practice this will go away.

Initially when I attached this keyboard to my 2018 iPad, it did nothing. Oh shoot, did I get a dud? I was so excited to play with this thing. No, it works fine. It turns out that I needed to run a software update on my iPad. After a slow update and reboot, I’m in business. Halfway through the reboot the keyboard just lit up… ahh.

I still don’t understand why the need for a hardware ‘globe’ button that enables the emoji keyboard. The iPad keyboard cover I was using prior has it as well. Why? It’s in the same place as the Fn key on a standard Mac laptop, so I instinctively hit it when I’m doing text manipulation. (I never realized how often I use Fn-Delete to remove characters in front of the cursor. It turns out that Control-D does the same thing, so I need to switch my muscle memory to that instead.) Lucky for me, I can use the keyboard settings in iOS to disable the silly globe key so it has no effect when pushed. Waste of space, but I can handle that.

The trackpad… wow there’s a trackpad for iOS! This is so delightful. It’s so natural for me to reach for the trackpad now instead of having to lift up my fingers and drag the screen around when I’m typing. It sounds silly, but it makes for such a nicer experience.

This week I was deploying a routine update to Heroku, but the app wouldn’t deploy. Here is the message that was displayed in the build log:

-----> Discovering process types
       Procfile declares types     -> release, sidekiq, web
       Default types for buildpack -> console, rake
-----> Compressing...
 !     Compiled slug size: 502.5M is too large (max is 500M).
 !     See: http://devcenter.heroku.com/articles/slug-size
 !     Push failed

Slug size is too large was the culprit. 502 MB! The app is decently sized but that seems way off to me. According to rails stats the app is about 58,243 lines of code. Not small, but certainly not huge. And these are mostly source code plain text files too so file size should not be an issue at this scale.

Investigating the Sizing

I used a basic du -sh * command to check the basic file sizes of the folders in the app’s repository to see what’s going on. Here’s the truncated output:

12M    app
352K    config
1.3M    db
376K    lib
332K    public
260M    node_modules
 78M    tmp
226M    vendor

So my app is relatively small, only about 17 MB. The rest of the file size comes from the various dependencies. By some standards, I don’t really even think this app has very many dependencies. There are a few dozen Ruby gems in the Gemfile and there’s nothing out of the ordinary here. Rails, Postgres, Nokogiri, etc. All things you would expect. (Although side note, the google-api-client gem is 61 MB on its own. Yikes, by far the largest gem in there. That’s something to work on for another day.) All of the gems are in vendor/ so the file size there checks out as reasonable.

The tmp directory is pretty big here too, but that’s ok. It is mostly filled with bootsnap caching files to make the app load quicker too. This would be an easy 78MB to clear out if it was the culprit, since I don’t really care how long the app takes to boot, but let’s reserve clearing it for later.

Even that massive node_modules directory isn’t really problem, since Heroku slugs are compressed before this slug size is calculated. Those dependencies compressed down are only about 80MB.

So there’s something up here. Even with my most conservative calculation the slug size should be max of around 300MB after compression. Which is still too big in my opinion for an app of this size, but still nowhere near the limit of 500 MB.

There is a nice Heroku plugin that allows you to download slugs from your builds. It’s a very handy way to download the exact slug from Heroku’s filestore on S3. I downloaded the slug in question to give it a look.

Using the same du command within the extracted slug files, there was one major diference:

961M    public

Whoa. 961MB in the downloaded and unzipped slug, but only 332K in my project’s repo itself. It turns out that inside the public/assets directory there were hundreds of asset build files dating back years. The app itself is a few years old and it looks like Heroku had stored the compiled assets in the slug for each release since the app was created. Yikes.

This is a known feature on Heroku and it is called the build cache. I was mistakenly under the impression that each app deploy would cause a new slug to be built, including the asset compilation. The build cache feature makes sense: why waste time re-downloading files and other dependencies on each build when they can be cached? But caching the output of asset creation until the end of time? That seems excessive. So let’s clean it up.

The Fix

It turns out there’s an easy fix when build cache is your problem. There’s another Heroku plugin for managing the repo that contains your app’s code. Within this plugin is the purge-cache utility, which clears the build cache.

Here’s how it works:

heroku plugins:install heroku-repo
heroku repo:purge_cache -a appname

I redeployed the app and that shaved off a few hundred MB from the slug size. We’re back in business here. But I still couldn’t get past that massive node_modules/ directory. That was 260MB of modules only used for compiling assets. Once the assets were compiled, those modules are no longer used. (This particular app does not use any node_modules at run time, they are all precompiled with webpack.)

There is a community-developed buildpack for Heroku called post-build-clean that gives us the ability to remove files from the slug after the build command finishes running. This is slightly different than the standard .slugignore file which won’t even allow you to use those files during the build phase. I installed the buildpack, added node_modules to the .slug-post-clean file and tried to deploy again.

This time around, it looks much more reasonable to me:

-----> POST BUILD CLEAN app detected
Removing directory node_modules/ from slug
-----> Discovering process types
       Procfile declares types     -> release, sidekiq, web
       Default types for buildpack -> console, rake
-----> Compressing...
       Done: 87.4M
-----> Launching...
 !     Release command declared: this new release will not be available until the command succeeds.
       Released v1019

For a medium-sized app 87 MB seems like a good size to me. The asset files continue to build up after the compilations step, but it’s going to be a while until I need to clear the cache again.

Marc Andreessen:

Every Western institution was unprepared for the coronavirus pandemic, despite many prior warnings. This monumental failure of institutional effectiveness will reverberate for the rest of the decade, but it’s not too early to ask why, and what we need to do about it.

Many of us would like to pin the cause on one political party or another, on one government or another. But the harsh reality is that it all failed — no Western country, or state, or city was prepared — and despite hard work and often extraordinary sacrifice by many people within these institutions. So the problem runs deeper than your favorite political opponent or your home nation.

Part of the problem is clearly foresight, a failure of imagination. But the other part of the problem is what we didn’t do in advance, and what we’re failing to do now. And that is a failure of action, and specifically our widespread inability to build. […]

Building isn’t easy, or we’d already be doing all this. We need to demand more of our political leaders, of our CEOs, our entrepreneurs, our investors. We need to demand more of our culture, of our society. And we need to demand more from one another. We’re all necessary, and we can all contribute, to building.

Every step of the way, to everyone around us, we should be asking the question, what are you building? What are you building directly, or helping other people to build, or teaching other people to build, or taking care of people who are building? If the work you’re doing isn’t either leading to something being built or taking care of people directly, we’ve failed you, and we need to get you into a position, an occupation, a career where you can contribute to building. There are always outstanding people in even the most broken systems — we need to get all the talent we can on the biggest problems we have, and on building the answers to those problems.

Interesting analysis of the sports world, specifically the in-stadium experience, in a post Covid-19 world by Alex Speier in the Boston Globe:

To maintain 6 feet on all sides, you'd likely need multiple empty seats and multiple rows between fans — some of which could potentially be offset by having, say, a family of four from one household sitting together in a block. Crowd composition could be altered further by profiling. Might teams discourage those at greater risk of serious complications from COVID-19 — those over 60, or those with preexisting conditions — from going to games? Would teams restrict tickets to — or feature different seating plans for — those who could document they had developed antibodies to the coronavirus?

Decontamination of stands will have to become a staple of stadium operations. Hand sanitizer will become omnipresent in concourses. Cleaning staffs would have to be vigilant about the "high-touch" areas of facilities — including railings (both in stands and on escalators) and elevator buttons.  Might there be a requirement for spectators to wear masks? If so, masks with team logos might replace caps or jerseys as the most frequently seen form of team apparel.

It's a quintessential part of the stadium experience: A hot dog passed from vendor to fan to fan to fan, with cash flowing back in the other direction. In all likelihood, that familiar ritual will be gone. "They'll have to have no stadium vendors," said Zimbalist. "They're not going to have people passing hot dogs down or passing anything down. That has to stop."

via Peter King's Football Morning in America

Some extraordinary photos of New York City this month by photographer Scott Medway on Exposure.