Near-Real-Time Activity System with Infinite Scroll
An activity feed that got slower the more you used it. Power users were being punished for showing up.
The problem
The old activity system fetched recursively across DynamoDB and SQL tables, enriching every object live, with zero pagination support. Every scroll was a fresh dig through history. The deeper someone’s history, the slower their own app got: success actively punishing itself.
What I had to work with
Two years of existing activity data had to migrate cleanly into whatever came next. No clean-slate luxury. And the bar was higher than the old system ever cleared: activities weren’t single events anymore, they were multi-step sequences. A Buy Order isn’t one thing. It’s Transfer Started, Transfer Completed, Order Placed, Order Filled, Units Allocated, each step meaningful on its own before it’s part of the whole.
What I built
I killed the live-lookup model entirely and replaced it with an event-driven pipeline.
Source DynamoDB tables (users, accounts, funding streams, transfers, orders) emit changes through DynamoDB Streams into activity-service, one Lambda run as a service. Its handleStream action decides (via filter rules defined in code, not hardcoded logic) which events actually deserve to become an activity step.
addActivityStep writes each one to the ActivityStep table, immutably and versioned: every activity type has versioned step definitions, so an old activity keeps its old shape while a new version can carry more. Nothing overwrites, nothing quietly disappears.
The ActivityStep table’s own stream triggers processActivityStep, which rolls steps up into a parent in the Activity table, and a parent only finalizes once it hits a terminal state. And because the real world doesn’t deliver events in a tidy sequence, I built for that explicitly: a step arriving out of order gets saved on its own, but the parent activity holds. It doesn’t update until the sequence resolves itself. No corrupted state. No silent data loss. Just a system that waits until it’s actually sure.
When an activity changes in a way search needs to know about, it’s marked syncStatus: PENDING and dropped onto SyncServiceQ. sync-service’s syncActivity pushes it to Elasticsearch and marks it synced, and a scheduled syncPendingActivities sweep catches anything that slipped through.
The mobile app talks to Elasticsearch through AWS AppSync, and pagination plus scalable search come baked in. No custom backend logic to write, maintain, or eventually regret. If Elasticsearch is unavailable, the AppSync pipeline falls back to the Activity table’s indexes: search gets limited, but the feed stays up.
What it cost
Backfilling wasn’t a copy-paste job. Two years of legacy data had to be pushed through the new step-based model and its presets before it could exist in the new world. That migration was practically its own project living inside this one. Every legacy record went into ActivityStep as a single-step V0 activity, in batches, and the pipeline built the Activities on its own. Loading it all took a few hours.
Where it landed
An async, horizontally scalable system that doesn’t slow down as people use it more, which was the entire point. Infinite scroll, powered by Elasticsearch, with a DynamoDB fallback sitting underneath in case it’s ever needed.