Support hero

We have two types of rotations for engineering:

Support Hero - first line responder to customer questions and bug reports
Secondary on-call - responsible for prioritizing and solving escalated issues, Sentry errors and alerts

When someone is Support Hero, they are also the secondary on-call person for their team

1. Support Hero

Every week, we assign one person to be the "Support Hero." If this is you this week, congratulations! Support hero is an intense but super fun week where you get to talk to a bunch of users, get the satisfaction of helping them out, and contribute to a lot of different parts of our system.

Expectations

All other work takes a back seat while you're doing support, so don't plan on doing any 'normal' work. Furthermore expect that you will need a day or two the week after to follow-up on the less urgent issues. Please don't hand off too many things to the next support hero and make the start of their week more stressful.

You should work your 'normal' hours during this week. There will likely be more issues than you'll have time to fix so be sure to prioritise. If there's an important and urgent issue near the end of your day, hand it off to someone on the other side of the Atlantic.

If you are planning on taking a day off or you won't be available, please find someone to swap with and update the rotation on PagerDuty. Be sure to schedule an override for both swaps and do not alter the rotation order to avoid affecting other members.

Rotation

You can view the Support Hero rotation in PagerDuty here.

Channels

There are a couple of channels that customer requests come in so make sure you keep an eye on all of them, but most support tickets will come into Zendesk. Beyond Zendesk, you should keep an eye on:

PostHog Users's Slack, specifically #community-support. Users who post in #general or elsewhere should be instructed to post in #community-support
Sentry issues, either directly or in #sentry in our main Slack.

Zendesk will also get populated with new issues from people outside the PostHog organization on the posthog and posthog.com repos, as well as with Squeak questions. These tickets will come with links to the issue or Squeak so you can answer them in the appropriate platform, rather than on Zendesk itself.

Communication

As an engineer, when a question comes in your first instinct is to give them an answer as quickly as possible. That means we often forget pleasantries, or will ignore a question until we've found the answer. Hence the following guidelines:

Always respond to a question within a reasonable timeframe during your working day (<15 minutes is great, <1 hour is okay, anything over a day is too late)
- If you're ready to look into the issue and you think it might take a while/require a fix, just mention that and say you'll get back to them
- If you have no idea how to answer or fix their issue, @mention someone who does
Start your response with Hey [insert name], ... and make sure you're polite, not everyone you talk to is an engineer and as accepting of terse messages
- If it's an email, make sure you format your message as an email and only send a single message, not multiple
Follow up!
Housekeeping. Once a customer issue/question has been addressed, close the ticket on Zendesk to make it easy to identify outstanding conversations.

To help manage users' expectations, you might find it useful to share a message in #community-support when you are signing on and off for the day.

Prioritizing requests

(To figure out how much a customer is paying, go to revenue.posthog.net (internal), click on the most recent month and command + F for the company name)

Respond to and debug issues for Priority customers (customers on Scale and Enterprise, either current or in our sales process, plus any high-paying Cloud customers)
Respond to and debug issues for Subscriber customers (paying subscribers on Cloud, usually paying $1-500/month)
Respond to and debug issues for Community users (all other free Open Source or free Cloud users)
Fix issues, create PRs

Triaging issues to the relevant small team

As the support hero you should triage any bugs/feature requests that you don't deal with to the relevant small team. They are ultimately responsible for handling bugs and feature requests for their area of the product.

Categorizing requests

It's really valuable for us to understand what types of requests we've had so we can prioritize our investments in certain areas and work out if we're making things better for our users (e.g. we use this as a measure of how easy it is to deploy PostHog).

When you initially respond to an issue in Zendesk add a "tag" with the following categories:

User experience (confusing/unclear UX).
Docs confusion either missing or confusing.
App Performance
Ingestion (either problems and not working or performance).
Insights
Data integrity
Deployments/Setup
Deployments/Upgrading
Deployments/Maintenance
Non-self-serve for requests that need to be processed manually (e.g. removing events).
Billing
Feature request
Bug functionality bugs, something is broken.

If something falls into two categories, but predominantly one, just tag the one you think is most relevant. If the ticket covers multiple topics, tag with all the relevant tags.

If a ticket doesn't fit a category correctly, we might need to update our tags -- open a PR to edit this page.

Escalating issues

You should always try to figure out the issues customers are having by yourself before escalating. This means gathering more information from the customer, reproducing the issue and ideally fixing the issue yourself by creating a PR. Only if you are completely swamped, you really don't have any knowledge about the area (most common with deployment questions) or you can't figure out what's going on should you escalate to secondary on call.

How to help customers

The reason it's so great to have engineers do support is that you can actually help customers solve their issues, rather than just escalating it. Hence you should aim to go deep and actually solve people's issues, whether that involves going deep on our functionality or spending half a day writing a PR to fix someone's issue
On app.posthog.com, you can log in as a user to help debug their issues.
- Do this by going to https://app.posthog.com/admin/posthog/user/, finding the relevant user and clicking 'log in as them'
- To go back to your old user, just log out
- If they have asked for help it is safe to assume they've given permission for you to log in as them.
- You can also check to see sentry errors tied to the user via the user.username parameter for e.g. for test@posthog.com
When trying to debug an issue with a customer and it's not immediately obvious, it's usually much faster to do a Zoom session. You also tend to get other useful product feedback.
When dealing with slowness, ask users to send a screenshot of their "System Status" page (under settings)
- If they have a lot of volume and they're still on Postgres they should probably upgrade to Clickhouse
Sometimes questions will have been asked earlier in the User's Slack so it's worth searching through that if you're not sure.

Debugging deployments

See the handbook page Deployments Support and user-facing Deployments Troubleshooting page.

Ingestion issues

If a user is sending events to PostHog and these are not getting ingested, despite them receiving a success response, try the following:

Check if the app/plugin server is alive and healthy (suggest a restart if not - this is safe)
Ask if they have Sentry set up and see any errors
If Sentry is not available, tell them to connect to ClickHouse and query for the columns error_location and error on the table events_dead_letter_queue

Reviewing new apps

From time to time, customers will request to get their apps added to PostHog Cloud, based on this tutorial. When this happens, do the following:

Review the app: check it doesn't do anything dangerous, like making an arbitrary number of requests, or attempt to DDOS some server.
Ensure it has a logo.png file
Fork their GitHub project
Add the forked project to our Integrations Repository
Tell the marketing team about this new integration
Install it on Cloud, and make it global

Zendesk

We use Zendesk Support as our internal platform to manage support tickets. This ensures that we don't miss anyone, especially when their request is passed from one person to another at PostHog, or if they message us over the weekend.

Zendesk allows us to manage all our customer conversations in one place and reply through Slack or email. We also use Help if you want to easily create and manage Zendesk tickets directly from inside the PostHog Users Slack.

This ensures that we don't miss anyone, especially when their request is passed from one person to another at PostHog, or if they message us over the weekend. You should have received an invite to join both Zendesk and Help as part of onboarding - if you didn't, ask Grace.

How to access Zendesk

You can access the app via posthoghelp.zendesk.com.

The first time you sign into Zendesk, please make sure you include your name and profile picture so our users know who they are chatting to!

How to use Zendesk

You’ll spend most of your time in the Views pane, where you’ll find all tickets divided into different lists depending on who they are assigned to, and whether they have been solved or not.

There are a few different ways that support tickets can be created in Zendesk:

Slack

User Slack

Support requests posted in #community-support are monitored by the Support Hero, who then creates tickets in Zendesk using the Zendesk app. Once created, the Support Hero will triage the request in either Zendesk or directly in Slack via Help, and assign it a priority and assignee.

Dedicated Support Channels

Our Customer Success team monitors these channels and creates tickets in Zendesk via a direct Slack integration, and will prioritize and assign them accordingly.

Email

Emails sent to hey@posthog.com automatically create tickets in Zendesk, which are prioritized and delegated by the Support Hero.

Goal of the Support Hero person on duty is to:

Keep the quantity of items in the Unassigned view as small as possible by assigning tickets to yourself as Support Hero, or in a minority of cases, another team member who is better suited to solve the ticket
Make sure all the conversations get a response in a reasonable time (we don’t want to drop customer requests on the floor)
Provide actionable information as Note for all the items you are handing over to someone else (note that you should follow-up on most requests yourself after your shift ends rather than assigning them to the next support hero).

How to deal with spam, marketing, partnership proposals, etc.

Like every other email address in this world, hey@ gets quite a bit of spam (and we reroute this to Zendesk). When this happens, simply mark the conversation as closed.

For marketing, partnership proposals or anything like that, please post in Slack in #team-marketing before taking an action.

Squeak!

Squeak! is a community curation toolkit created by the Website & Docs team. The ultimate goal of Squeak! is to support our customers and make our community self-sustaining.

Q&A.js by Squeak!

At the end of every page in the docs and handbook is a form where visitors can ask questions about the content of that page. (It also appears on the pricing page, and will be used in other places in the future.) This is an embedded JavaScript snippet we call Q&A.js, powered by Squeak!

Q&A.js should help reduce the load of the support hero, but you'll need to make sure questions are getting answered accurately – and in a timely manner.

Answering questions using Squeak!

When a question is posted, a notification is sent to the #squeak-ping channel in Slack. You can answer a question directly on the page where it was asked. When a reply is posted, the person who asked the question will receive an email notification.

The first time you answer a question, you'll need to create a Squeak! account. (You'll be prompted to do this after answering a question, as posting/responding requires authentication.)

Important: Do not sign up directly on squeak.cloud. Start by answering a question on the site - this creates an account within PostHog's Squeak! Cloud account. (As an alternate, you can ask an existing admin to invite you.) (Signing up directly on squeak.cloud creates a new account for you to use Squeak! on another site, which isn't what we want.)

Ask in #squeak-ping to be upgraded to a moderator. This will also give you access to the admin panel hosted on squeak.cloud to manage questions with moderator controls.

Note: Squeak! currently uses a separate authentication system from PostHog Cloud. There are plans to support other types of authentication so a visitor doesn't have to create a separate account for asking questions.

2. Secondary on-call

Every team has a Secondary on-call rotation. Unlike support hero, you are still expected to do feature work. During the week that you are on-call, you are responsible for prioritizing and solving escalated issues, sentry errors and alerts that happen within your team. It also means helping out the support hero where necessary.

Schedules

PagerDuty doesn't let us have a rotation that automatically selects the person that is support hero to also be the secondary on-call for their team. This means we'll occasionally need to manually shuffle the schedule around.