Sprint Oct 6 - Oct 17

[MichaelKutsch-ph]

Sprint Planning

List of incidents + any action items that haven't been scheduled yet

Team sprint planning

For your team sprint planning copy this template into a comment below for each team.

# Team ___

**Support hero:** ___
**Off during the sprint:** ___

## Quarter goals

1. Quarter goals, status (red/yellow/green) and action points if yellow/red

## Retro

### Tasks

<!-- Grab the high and low priority items from last time and add whether that item was completed or not -->

## Plan

### High priority

-

### Low priority / side quests

-

[MichaelKutsch-ph]

Team Infra

Support hero: @MichaelKutsch-ph @frankh

Retro

High priority

✅ - persons migration: dry runs and prepare actual migration @danielxnj @bciaraldi
🟡 - LogHog v1 @frankh

• Security @Piccirello
  ✅ - WIZ Contract
  🟡 - PoC PerfectScale @danielxnj
  🟡 - Clickhouse EKS: finish minimum setup so that we can install clickhouse operator @MichaelKutsch-ph - bootstrapping an EKS cluster with argo works, currently setting up karpenter, monitoring, external-dns so we can hook up clickhouse operator
  ⚪ - Internal helm charts repo migration @bciaraldi
  ✅ - Enhance geoip protections at the edge @bciaraldi - Live in the EU

✅=finished 🟡=in progress 🔴=won't finish ⚪=not started

OKR Infra

• 🪵 LogHog @frankh
• ↔ Managed reverse proxy for everyone - Istio version @danielxnj
• 🔒DevEx friendly Secrets Management @danielxnj @Piccirello
• 📄 Make bigger chart adjustments less risky @bciaraldi
• 🧹 Clean AWS Account and kubernetes bootstrapping @MichaelKutsch-ph
• security goals @Piccirello @MichaelKutsch-ph

Side quests / other work

• 💽 Help team ingestion make persons split happening

High priority

• Support posthog persons migration when needed @danielxnj @bciaraldi
• Managed reverse proxy in dev using Istio Gateway @danielxnj
• Finish geoip in the EU/US (testing in Surveys, admin and other apps) @bciaraldi
• Internal helm charts repo migration @bciaraldi
• Wiz @Piccirello
• Hardening our security context @Piccirello
• EKS module to work out of the box and test it with clickhouse @MichaelKutsch-ph
• Argocd audit @langesven
• Github Webhooks check @langesven

Low priority / side quests

[tomasfarias]

Team Batch Exports

Quarter goals

• Export any data warehouse view
• Any realtime destination can be powered by batch exports

Retro

• Databricks destination for batch exports @rossgray

  • Backend work / Temporal workflow
  • Frontend work
  • Rollout

• Support new billing approach @tomasfarias

  • Fail runs if over the quota limit (first metric only then drop as follow up PR)
  • Add failed billing metrics

• Get metrics exported into appmetric2 table (like we do for hog functions) @tomasfarias

  • Run metrics
  • Rows exported metrics
  • Metric in the frontend

• PostgreSQL/Redshift batch export @tomasfarias

  • Migration to new pipeline
  • COPY support for Redshift

High priority

• Databricks destination for batch exports @rossgray

  • Backend work / Temporal workflow
  • Frontend work
  • Rollout

• Support new billing approach @tomasfarias

  • Fail runs if over the quota limit (first metric only then drop as follow up PR)
  • Add failed billing metrics

• Get metrics exported into appmetric2 table (like we do for hog functions) @tomasfarias

  • Run metrics
  • Rows exported metrics
  • Metric in the frontend

• PostgreSQL/Redshift batch export @tomasfarias

  • Redshift Migration to new pipeline
  • PostgreSQL Migration to new pipeline
  • COPY support for Redshift

• Investigate heartbeat timeouts in Temporal @tomasfarias / @rossgray

  • Make workflows robust against being preemtively cancelled.

[havenbarnes]

Team Massaging an MVP

Vacation

• Marcus 5d
• Myke 1d
• Abe 1d

OKR

• Grow native Messaging
• Workflows as a platform
• Destination ergonomics
• AI-powered workflows

Retro

High priority

• Roll out messaging to more customers @havenbarnes / @abheek9
  • Address highest priority bugs / feedback as they come in
• Have our first behavioral cohort working @meikelmosby
• Finish Spike - kafka based queue delays @MarconLP
• finish work to store message attachments in private s3 @havenbarnes
• pull trigger which email provider (mailjet vs aws) @PostHog/team-messaging

Low priority

• Release incoming webhooks to GA EA
• finish up docs for messaging
• refactor the cdp docs to follow the other products @meikelmosby @abheek9
• improve max-ai integration for hog code generation (back burner stuff) @meikelmosby

High priority

• Early access release for messaging
• Fully transfer to SES @havenbarnes
• Behavioral cohort membership tracking @meikelmosby
  • Dani working on CH part
  • Read membership updates from Kafka -> write to PG
• Address memory leak in hog kafka workers @benjackwhite
• finish work to store message attachments in private s3 @havenbarnes
• load test volume of emails @havenbarnes
• Have a working MVP for a review cycle @MarconLP
• Automatically pull data from Deel @MarconLP
• Implement keeper tests @MarconLP

Low priority

• Growth review for CDP @abheek9
• Monitor EA for incoming webhooks, prep for GA @havenbarnes
• Add campaign templates to messaging @meikelmosby (if time allows)
• Continue with Messaging release checklist items @team

[mariusandra]

Team Users Experienceing the Platform

New quarter! 👋

OKR

• Build a new “Search / Command palette / Shortcuts / New tab” experience
  • 🟡 Move project tree files into the “new tab”
  • 🟡 Search can access everything and will replace the existing command palette
  • 🟡 Integrate directly with Max
• PostHog doesn’t feel confusing
  • ⚪ Remove right sidebar, move into new tab experience
  • 🟡 Clean up the last 20% of the tabs/actions-panel/etc work
• Bonus: Filters, electron, sparking joy
  • ⚪ Improve the taxonomic filter
  • ⚪ Electron app now that we have tabs

🟢=finished 🟡=started on 🔴=won't finish ⚪=not started

Retro

Adam was off 2nd week (this week). It was a slow burn. Updates to the new tab page, cleaning up UX issues as we found them.

• Make our UX spark joy by finishing all in flight projects
  • SQL editor untabbing
    • ✅ Cleanup and whatever remains
    • 🟡 Integrate it nicely into product analytics/insights
  • New tab scene
    • ✅ Add project items
    • 🟡 Add AI features (max in the tab)
  • Clean up, clean up, clean up
    • ⚪ Ongoing

Team

Nothing special this sprint

High priority

Will complete in full detail Monday once Adam is back.

• New tab
  ⚪ Think through the categories and items
  ⚪ Different sorting options for project items
  🟢 Add shortcuts here
  ⚪ Open folders as index pages
  ⚪ Personalise it: "your" recently used files, "your" recently clicked actions. Track your actions somehow. Activity logs?
  ⚪ How to integrate searching for ClickHouse data?
  ⚪ Integrate Max AI better into the question form.

Lower priority

⚪ Deletes into project tree.
⚪ Lushious app tooltips

[jurajmajerik]

Team Experiments 🧪

Retro

Juraj

• Timeseries: finished fingerprinting and jobs for regular + scheduled metrics. Will release the MVP to first customer
• Feature improvements: UI tweaks, default labels, “persist flag” field + experiment
• Tech interviews, hiring spec, Q4 planning

Anders

• Funnel chart based on experiment results is out behind a feature flag. Still doing some testing and fixing / tweaking things. On track to be released to all within this sprint (by the end of the week)
• Started on a new query builder that uses “plain HogQL + placeholders”, much nicer to read!
• Other fixes
  • Preserve metric when switching metric type in the form
  • Remove unused field from experiment query
  • Add option to ignore zeros in outlier handling (need to update docs, working on this)
• Performance work
  • Testing out map aggregations instead of a join. So far it seems to increase memory consumption.

Rodrigo

• Support Hero!
• Not a lot of actionable items, except for the query performance issue.
• Create Experiment:
  • Merged the base structure and the first panel on feat(experiments): unified create experiment base structure #38831, extracted from the main draft on feat(experiments): unified create experiment form #38767
  • Complete refactor of the public interface of the Metric Modal so it can be reused in other places (like the create form) on refactor(experiments): remove experiment dependencies from metrics dialog #38071 and refactor(experiments): metric modal save cleanup. #38756
• Breakdowns
  • Integrated a custom “Add Breakdown” button to the metric header on feat(experiments): add breakdown button to metric header. #38568
• Q4 Goals
  • First draft of the “Experiment Agent” RFC at Basic toolbar with actions #868. Still a work in progress, but the core idea is there.

Annika

• Not working on experiments at the moment, focussing on my other teams for now 💆‍♀️

---

Sprint goals

Juraj

• Timeseries
  • Backfill – recalculate entire timeseries on demand
  • Optimizations – process in batches, make sure the system scales when released wide
  • Make the UI nicer
• Array <> Experiments integration – start exploration

Anders

• Work on performance. As part of that, refactor the query runner to use the new query builder. Much nicer to work with and much less code.
• Kick off some threads with the feature flags team on using “device_id” as identifier for feature flags and running multiple experiments on the same feature flag.

Rodrigo

• Support! (remainder of this week)
• Primary Goal: Launch Create Experiment – continue splitting code into small PRs for review and merge to launch an experiment as soon as possible.
• Secondary Goal: Complete Breakdowns UI – continue working on the UI integration to send the breakdown information as part of a query. Start working on the query modifications.
• Side Quests:
  • Experiment Agent RFC – finish the alternative approaches and the project management section.
  • Freeze Legacy Experiment – isolate the code for Legacy experiments so it’s easier to modify and test new things on the experiment view. Draft PR: chore(experiments): freeze legacy experiment view #38800

[pl]

Team Ingestion

• Support: @jose-sequeira

Retro

High priority

• Persons DB
  • 🟢 Migrate dev over @nickbest-ph
  • 🟢 Update runbooks for the prod migration @nickbest-ph
  • 🟡 Set up infrastructure for the prod migration (pgbouncer, etc) @nickbest-ph
  • 🔴 Implement the K8s jobs for the new persons db migrations @nickbest-ph
  • 🟢 Ensure sqlx migrations work @nickbest-ph
  • 🟢 Schedule the maintenance windows for prod, include statuspage updates @nickbest-ph
  • 🟢 Ensure we have a solid Postgres recovery plan @nickbest-ph
• Deduplicator
  • 🟢 Implement the deduplication algorithm @jose-sequeira @eli-r-ph
  • 🟢 Implement the deduplicator producer @jose-sequeira @eli-r-ph
• LLM Analytics Ingestion
  • 🟢 Plan the implementation of the LLM Analytics capture @pl
• plugin-server refactoring
  • 🟢 Clean up redirects and DLQ handling @pl
  • 🔴 Switch cookieless to use the timestamp from the headers @pl
  • 🔴 Remove timestamp normalization from plugin-server @pl
  • 🔴 Add DLQ in Blobby V2 @pl
• 10x Planning
  • 🔴 Design the support sharing session @nickbest-ph
  • 🟢 Gather hackathon ideas @pl

Incident follow-up

None

Support follow-up

None

Side quests

• Rust profiling for kafka deduplicator @eli-r-ph
• Roll out the shufflehog topic for propdefs @eli-r-ph

OKRs

• 🟢 Implement the LLM Analytics Pipeline
• 🟢 Define the event ingestion boundaries and enforce them
• 🟢 Prototype Persons API
• 🟢 Prototype multi-threaded ingestion
• 🟢 Do at least one thing that improves data quality for customers

High priority

• Persons DB
  • Production migrated @nickbest-ph
• Deduplicator
  • Solve the performance issues @jose-sequeira @eli-r-ph
  • Dry run on production @eli-r-ph @jose-sequeira
• LLM Analytics Ingestion
  • Implement acceptance tests @pl
  • Implement phase 1 of LLMA capture @pl
• plugin-server refactoring
  • Centralize ingestion warnings @pl
  • Migrate more steps to the new framework @pl
• 10x Planning
  • Design the support sharing session @nickbest-ph
  • Schedule the offsite
• Rate Limits
  • Start outlining the ingestion limits @eli-r-ph @jose-sequeira

Incident follow-up

None

Support follow-up

None

Side quests

• Investigate failed merges in the EU?
• Propdefs lag?

[pauldambra]

Team Re(ar quarter of the year)play

Support hero:
@veryayskiy

OKRs 🟡

watch this space

Retro

• SDK lazy loading done done ❌ ➡️ @pauldambra
  • second attempt
• export to gif/video - make it right ➡️ @veryayskiy
  • now about using it in all the places
• dynamic retention @TueHaulund
  • just the product upsell pieces
• started on hard delete for recordings compliance
• heatmaps @veryayskiy while it's inflated in his head
• first draft of app state logging out for customer to test
• some progress on memory leak problem

PRs

bugfixes and support

• feat: hide hidden properties when listing them (posthog, @pauldambra)
• fix(replay): disable playlist actions if no recordings in the list (posthog, @veryayskiy)
• fix: don't mangle to dollar in any build (posthog-js, @pauldambra)
• Update steps for session replay installation guide (posthog.com, @pauldambra)
• fix: no billing api calls in selfhosted (posthog, @pauldambra)
• chore: remove similar recordings feture (posthog, @veryayskiy)
• fix: maybe we skip meta patching for cached resources in replay playback (posthog, @pauldambra)
• fix: url fallback should not consider hash params (posthog-js, @pauldambra)
• fix: maybe debounced loading is causing the snapshot to flicker (posthog, @pauldambra)
• chore: flapping execution time tests (posthog-js, @pauldambra)
• fix: default person display properties (posthog, @pauldambra)
• fix: try to fix flappy snapshot by waiting (posthog, @pauldambra)
• chore: fix snapshot (posthog, @pauldambra)
• fix: remove realtime polling code from replay playback (posthog, @pauldambra)
• fix: toolbar visibility detection (posthog, @pauldambra)
• fix: poll for notifications only when visible (posthog, @pauldambra)
• fix: no new prompt for replay playlists (posthog, @pauldambra)
• chore: added db migration to get rid of legacy retention option (posthog, @TueHaulund)
• fix: nicer replay chrome hover behaviour (posthog, @pauldambra)
• fix: make sure older migrations can still run (posthog, @TueHaulund)
• fix: self hosted caddy host should have been quoted (posthog, @pauldambra)

heatmaps

• feat(heatmaps): support scroll depth in the app (posthog, @veryayskiy)
• fix: scroll update timer too slow (posthog, @pauldambra)

playlists

• refactor: decouple saved playlists logic (posthog, @veryayskiy)

comments

• feat: event when comments are hovered in replay (posthog, @pauldambra)
• feat: notebook replay comments can just be comments (posthog, @pauldambra)

hard delete

• fix: make sure to trigger Temporal deployment when delete-recordings … (posthog, @TueHaulund)
• fix: use simpler query to look up session IDs for set of distinct IDs (posthog, @TueHaulund)
• fix: load metric container via activity instead of via workflow (posthog, @TueHaulund)
• feat: added new Temporal workflow for deleting recordings (posthog, @TueHaulund)
• fix: use crontab instead of relative scheduling (posthog, @TueHaulund)

memory leak

• fix: potential memory leak in xhr network plugin (posthog-js, @pauldambra)
• fix: multiple possible memory leaks (posthog, @pauldambra)
• fix: session replay scene leaves lots of detached DOM nodes (posthog, @pauldambra)

lazy load SDK

• chore: switch back to lazy loaded (posthog, @pauldambra)
• chore: opt back out of lazy loaded replay (posthog, @pauldambra)
• fix: buffer rotation while pending (posthog-js, @pauldambra)
• fix: 2 small conditional recording fixes (posthog-js, @pauldambra)
• fix: recording should stop on forced idle and (conditionally) start on return (posthog-js, @pauldambra)
• fix: over eager rename of onRemoteConfig (posthog-js, @pauldambra)
• feat: turn session replay lazy loading back on (posthog, @pauldambra)
• fix: replay config persistence when lazy loading (posthog-js, @pauldambra)

clipping

• replay screenshots and clips (posthog.com, @veryayskiy)

app state logging

• chore: try again with app state logging (posthog, @pauldambra)
• fix: app state logging diff performance (posthog-js, @pauldambra)
• chore: turn off app state logging (posthog, @pauldambra)
• feat: turn on app state logging for 10% of sessions (posthog, @pauldambra)
• feat: redux aware logger display (posthog, @pauldambra)

side quests

• feat: release replay hover mode to everyone (posthog, @pauldambra)
• feat: quick filters in replay url (posthog, @pauldambra)
• fix: wrap conversation view in a flag (posthog, @pauldambra)
• feat: show LLM events as a conversation (posthog, @pauldambra)
• feat: add ph-no-rageclick in playercontroller (posthog, @pauldambra)
• feat: allow ignoring rageclick by css selector (posthog-js, @pauldambra)

High priority

• hard delete @TueHaulund
  • got the basics in place, now smash it into prod
  • hard delete for metadata
• upselling configurable retention @TueHaulund
• make clipping visible @veryayskiy
  • need to be able to find them and share them easily
• Snappy wasm so we don’t decompress in the API @pauldambra

Side quests and lower priority

• redux/kea state logging @pauldambra
• replay RAM usage @pauldambra (talk to @mariusandra probs)
• replay shorts for fun and great good @veryayskiy
• comments / screenshots / clips - how do we pull them together @pauldambra @veryayskiy

[Twixes]

Team Max AI

Support hero: Alessandro & Denada
Off during the sprint: Alex 3d, Georgiy 3d, Alessandro 6d

Quarter goals

See team page.

Retro

Tasks

Optimize the collapsible blocks in notebooks @Twixes
Continue session summaries user interviews @sortafreel @Twixes
ChatGPT-ify Max (e.g. web search) @Twixes
Video confirmation for session summaries @sortafreel
New loop executor @skoob13
EU OpenAI endpoint @skoob13
Fix Redis not available in Dagster @skoob13
Resolve Temporal perf issues @skoob13
Edit existing dashboards @denakorita
Faster taxonomy exploration (parallelized) @denakorita
Fix deep research bugs for release @tatoalo @kappa90

Plan

High priority

Video confirmation for session summaries @sortafreel
Look through session recordings of people using session summaries @sortafreel @Twixes
Use realistic data in dev @Twixes
AI platform tools refactor @kappa90
Fix UI in single loop executor @skoob13
- Web search @Twixes
- Deep research out @kappa90
Improve reasoning/task updates UI @skoob13
Temporal-ize Max tools @skoob13
Edit existing dashboards @denakorita
Go through traces and errors! @everyone
First source of Non-Enterprise Search @tatoalo

Low priority / side quests

Put in Temporal profiling (distributed tracing) to sort Max delays @tatoalo

[daibhin]

Team Error Tracking

@hpouillot off second wee

Retro

✅ Q4 planning: new goals posted on https://posthog.com/teams/error-tracking
✅ Great week in Lisbon
🎉 Built v1 of better grouping through embeddings

OKR

• Go all in on AI
• Ship mobile error tracking
• Research deep dive on customers
• Increase trust in product
• Improving context on issues
• Stretch: Billing, spikes and alerts

High priority

• Android exception capture (part of [Megaissue] Mobile error tracking #38845) @marandaneto
  • SDK manual and autocapture
  • New raw frame type in cymbal (no need for line context)
• Roll out better grouping through embeddings as early access feature? (or everyone) @oliverb123
• Flutter exception handling ([Megaissue] Mobile error tracking #38845) @ioannisj with guidance and early feedback from @marandaneto
  • Flutter errors / uncaught exceptions
• @daibhin ignore our own exceptions in the PostHog SDK
• @daibhin integrations for LLM providers
• @daibhin decide on issues in ClickHouse approach
• @hpouillot React Native methods
• @fivestarspicy share growth review
• @fivestarspicy start research plan
• @ablaszkiewicz merge Nuxt integration
• @ablaszkiewicz v1 of breakdowns query

[arthurdedeus]

Team Customer analytics

Support hero: @arthurdedeus
Off during the sprint:

OKR

1. Default dashboard with customer base insights

   • Active users (DAU/WAU/MAU) and their trends over time
   • Signups
   • Paying customers
   • Free to paid conversion rate
   • Customer engagement/usage

2. Better individual customer visualization (for groups and people)

   • LLM traces
   • Exceptions
   • Support tickets

3. Answer questions with Max

   • I want to interview product engineers that are using product X, give me a list ranked by usage.
   • I am about to jump on a call with customer X about Y. What do I need to know about them?

Retro

High priority

• User interviews with PostHog product engineers 🟢
• Q4 objectives 🟢
• Customer analytics RFC 🟢
• Pin person properties 🟢
• Person session summarization 🟡
• Person LLM Traces 🟡

Low priority

• Scope data warehouse join customization ⚪️

Plan

High priority

• AI session summary button #38941
• LLM traces #38967
• Exceptions #38968

Low priority

• Usage metrics #39179

[Daesgar]

Team Clickhouse

AKA Team Make Computer Do Less

Retro:

High priority

• 🟢 Completed Propdef topic migration @fuziontech @tkaemming
• 🟡 Migration service @orian
  • Adding more engines supported by the service
• 🟢 Onboarding in London @fuziontech @sakce
• 🔴 EU coordinators in EKS @Daesgar

Low priority / side quests

• 🔴 Housekeeper @fuziontech

High priority

• Behavioral cohorts membership queries @Daesgar
• Move sharded tables' ingestion to ingestion layer @Daesgar
• Handbook PR to have a deletion policy @Daesgar
• Leave Endpoints frontend ready @sakce
• Catching up on load scheduling discussions to start SlopCop @sakce
• Increase EBS throughput to 2000 @fuziontech
  • Probably needs upgrading of the AWS Terraform modules we are using
• Get caught up and help with migrations @fuziontech
• Add support to add columns to the schema service @orian
• Find out what's the state of the schema in all CH nodes @orian

Low priority / side quests

• Test Iceberg writer though Warpstream's tableflow @fuziontech
• Column pruning on subqueries coming from HogQL @sakce
• squash some more 🐛s in the query_log and how queries appear @sakce

✅=finished 🟡=in progress 🔴=won't finish ⚪=not started

Questions / Discussion