How Reddit uses Sentry

For the sixth edition of Customer Stories we spoke to Andy Tuba, a Senior Software Engineer at Reddit. Reddit is a site that needs no introduction, but we’re gonna write one anyway because otherwise this section would just be blank. They bill themselves as the front page of the internet, and considering they’re the 8th most popular website in the world, that isn’t just marketing pablum. Some people spend their entire day at work on Reddit, instead of the site for whatever less popular company they work for.

(Editor’s note: This video literally contains more tuba than any film — long or short — since Mr. Holland’s Opus.)

Andy’s story

I’ve been in an employee at Reddit for going on two years now, but I’ve been a contibutor to the Reddit ecosystem for going on seven years. There are tons of third party extensions for Reddit: there’s mobile apps, there’s desktop browser extensions, there’s stylesheets; many, many ways to alter and expand upon Reddit’s own site. I’ve personally been a long-time contributor to the Reddit enhancement suite, which is what eventually led to me working here.

My goal as an engineer is to listen to the users, take a close look at how they’re using Reddit, learn more about what they want out of the site, and give it to them. I want it to be easy for users to find the communities they’re interested in, and then chat, comment, and enjoy all the cat pictures and discussions they like.

Deploying Code

When you want to deploy some code at Reddit to make improvements for those users, there’s a similar process across all our teams. After your code is reviewed, you go to the Slack channel that represents your team and you ask Harold the Robot, “Could I please have the conch?”

Harold might say, “You have to wait in line, but when it’s getting to be your turn I’ll tell you.”

Then once your turn arrives, he’ll say, “It’s your turn, you have the conch, you may deploy.”

Once you log into your service health dashboards, your error monitoring dashboards, and your server error logs, it’s time to roll out. We have a tool called the Rolling Pin that compiles and pushes to all the app servers for your service.

Monitoring That Code

Of course, I don’t want there to be bugs when I push code out, but there likely are going to be bugs. It’s pretty unavoidable. Two key ways I keep an eye on my code is:

• Self-monitoring
• Sentry

What I mean by self-monitoring is that as an engineer, and really just as a professional, there are days where you’ll feel sleep or cranky and really just not want to deal with anyone. When that happens to me, I dig into what’s causing it: did I not eat a good lunch? Did I not get enough sleep last night?

I ask myself if there’s something I can go do to improve how I’m feeling: take a walk around the block, find a snack, do a little meditation.

I set reminders for myself to go eat lunch by 1:30, to start wrapping up and packing to go home at 6. It does no one any good to push yourself beyond reason, that’s a great way to make mistakes and write a lot of bugs.

And with Sentry, what I really appreciate is that it’s collecting related errors into a single issue. On a site as big as Reddit we have a lot of errors coming, but usually they have the same root cause or maybe 50 same root causes. Those 50 different root causes might lead to 50,000,000 items, but when I log in to my dashboard and see there’s only 50 items to deal with that each happened a million times, it’s so much easier than trawling through every single bug report.

We also use Sentry’s Slack integration so that we’re pinged to the right channel whenever a new issue pops up. That’s a lot better than staring at a dashboard all day.

The technical Details

Sentry’s Slack integration does far more than just ping you whenever a new error is discovered. You can:

• Send specific kinds of errors to specific #channels based on your own alert settings. Should every billing error immediately ping #billing? Great. Should an error that’s thrown a hundred times in a minute notify #nononononononono? Done.

• Assign issues to Sentry teams or users from Slack. Select who should tackle the problem directly in the Slack notification, and it’ll be assigned to them synchronously in Sentry to kick off the resolution workflow.

• Resolve issues directly within Slack. When the issue is fixed, mark it as resolved from the same Slack notification that made you aware of the problem.

As noted above, once you’ve activated Slack for your organization, you can configure your notification settings so you’re alerted at specific Slack channels or workspaces depending on the type and severity of the error.

How does is work?

First, Confirm your Slack workspace is configured for your Sentry organization by clicking on All Integrations from within any of your Project Settings.

Click Configure and you’ll see the available Slack workspaces for your project. From this page, you can click on Add an Alert Rule for this project to go directly to your Alert Rule settings.

You can also access Alerts from your Project Settings. There, you can configure when notifications are sent to your Slack workspace(s). You can route notifications in a few ways:

• To a specific channel in your Slack workspace
• To multiple channels in your Slack workspace
• To multiple Slack workspaces

Click New Alert Rule to configure a new Alert.

For any existing Alert Rules you previously configured, you can edit the rule to update the Slack channel routing:

When you click Edit Rule, under Take these actions you can specify your Slack channel(s):

After selecting Send a notification to the {workspace} Slack workspace to {channel} and include tags {tags}, you can specify the workspace, channel(s), and tags you’d like to include with your Alert Rule.

You can add Alert Rules routing to as many Slack channels as you’d like!

From the Slack notification, you can use the Resolve, Ignore, or Assign buttons to update the Issue in Sentry.