All Work
E-CommerceAI Platform2024

Scaling a Creator Marketplace: 100M-Row Attribution + AI Pipelines

  • Node.js
  • PostgreSQL
  • pgvector
  • AWS Aurora
  • Redis Streams
  • BullMQ
  • Prisma
  • Stripe Connect
  • OpenAI Whisper
  • GPT-4o
  • Gemini
  • Shopify API
  • Grafana

The challenge

Creators are paid commissions on attributed sales, so attribution is a payments problem. Heading into 4x growth, the service ran at ~100% CPU, DLQs hit six figures, and event tables outgrew the DB.

What we built

A first-party JS pixel + Shopify webhook attribution engine on Redis Streams into Postgres; migration to AWS Aurora; plus an AI pipeline: Whisper, GPT-4o matching, Gemini analysis, pgvector search.

Systems delivered

  • First-party attribution engine
  • AWS Aurora migration
  • Postgres performance engineering
  • AI creative intelligence pipeline
  • pgvector semantic search & matchmaking
  • Stripe Connect commission payouts

Outcome

Events table shrunk 146GB to 10GB with zero data loss on ~100M visitor rows, served reliably while prepared for 4x traffic growth. 3 AI providers in production.

Have a similar product challenge? Let's talk about the right technical path.

Discuss a Similar Build
Scaling a Creator Marketplace: 100M-Row Attribution + AI Pipelines | Techlusion Work