Finvo

📧 Broadcast Email Deduplication Guide

Kopier som prompt til andre apps

# BROADCAST EMAIL DEDUPLICATION GUIDE
# SĂ„dan sikrer du at der KUN sendes 1 email pr. unik modtager
# Sidst opdateret: 2026-04-02

=====================================================================
PROBLEMET
=====================================================================

Broadcast-kampagner afsendes via et cron-job hvert X minut.
Uden beskyttelse kan det samme kald (eller to overlappende kald)
sende den samme email to gange.

Risici:
- [HØJ]    Duplikerede modtagere i BroadcastRecipient tabellen
- [MEDIUM] Race condition: to overlappende cron-kald behandler samme modtager
- [HØJ]    Retry uden statustjek sender emails der allerede er sendt


=====================================================================
LØSNINGEN — 3 BESKYTTELSESLAG
=====================================================================

LAG 1: In-memory deduplication
  → Dedupliker alle recipients i memory FØR afsendelse (email_normalized)

LAG 2: Claim-before-send (UDEN re-fetch)
  → Sét sent: true FØR Brevo-kald. Revert til sent: false ved fejl.
  → VIGTIGT: Lav IKKE et ekstra .filter({ id }) re-fetch per recipient —
    det koster 1 ekstra DB API-kald per email og rammer Base44's rate limit (429).
    In-memory deduplication (LAG 1) er tilstrĂŠkkelig beskyttelse mod dobbelt-send.

LAG 3: Rate limit + cron interval
  → 50 emails per kald, 400ms delay, 2 minutters cron interval


=====================================================================
ENTITETS-FLOW
=====================================================================

BroadcastCampaign (status: draft → sending → sent)
    ↓
BroadcastRecipient (will_receive: true, sent: false → sent: true)
    ↓
Brevo API (afsender selve emailen)

BroadcastCampaign felter:
  status: 'draft' | 'sending' | 'sent'
  sent_count: number
  failed_count: number
  will_receive_count: number
  sending_started_at: datetime
  sent_at: datetime

BroadcastRecipient felter:
  campaign_id: string
  email: string
  email_normalized: string   ← KRITISK (lowercase/trimmed)
  will_receive: boolean
  sent: boolean
  failed: boolean
  sent_at: datetime
  error_message: string


=====================================================================
LAG 1: IN-MEMORY DEDUPLICATION (kode)
=====================================================================

// Hent ALLE unsent recipients (pagineret — VIGTIGT!)
const allRecipients = [];
let skip = 0;
while (true) {
  const batch = await base44.asServiceRole.entities.BroadcastRecipient.filter(
    { campaign_id: campaign.id, will_receive: true, sent: false },
    '-created_date', 500, skip
  );
  if (!batch || batch.length === 0) break;
  allRecipients.push(...batch);
  if (batch.length < 500) break;
  skip += 500;
}

// Dedupliker baseret pÄ email_normalized
const seenEmails = new Set();
const recipients = [];
const duplicates = [];

for (const r of allRecipients) {
  const key = (r.email_normalized || r.email || '').toLowerCase().trim();
  if (!key || seenEmails.has(key)) { duplicates.push(r); continue; }
  seenEmails.add(key);
  recipients.push(r);
}

// Marker duplikater som sent
if (duplicates.length > 0) {
  await Promise.all(duplicates.map(r =>
    base44.asServiceRole.entities.BroadcastRecipient.update(r.id, {
      sent: true,
      sent_at: new Date().toISOString(),
      error_message: 'Skipped: duplicate email address'
    })
  ));
}

FALDGRUBER:
! Brug email_normalized (lowercase/trimmed) som nĂžgle — IKKE rĂ„ email
! Husk paginering — stop for tidligt og duplikater overlever pĂ„ tvĂŠrs af sider


=====================================================================
LAG 2: CLAIM-BEFORE-SEND (kode) — UDEN re-fetch
=====================================================================

⚠ KRITISK: Lav IKKE et .filter({ id: recipient.id }) re-fetch per recipient!
   Det koster 1 ekstra Base44 API-kald per email → 50 ekstra kald per run →
   rammer hurtigt Base44's rate limit (429) og stopper hele kampagnen.

for (const recipient of recipientsToSend) {

  // CLAIM — sét sent: true FØR Brevo-kald (ingen re-fetch)
  await base44.asServiceRole.entities.BroadcastRecipient.update(recipient.id, {
    sent: true,
    sent_at: new Date().toISOString(),
    error_message: 'Sending...'
  });

  try {
    await sendViaBrevo(recipient, campaign);

    // SUCCESS — bekréft og ryd placeholder
    await base44.asServiceRole.entities.BroadcastRecipient.update(recipient.id, {
      sent: true, failed: false, error_message: null
    });

  } catch (error) {
    // FEJL — REVERT claim sĂ„ nĂŠste kald kan retry
    await base44.asServiceRole.entities.BroadcastRecipient.update(recipient.id, {
      sent: false, failed: true, error_message: error.message
    });
  }
}

FALDGRUBER:
! Glemmer du at revertere sent: false ved fejl, hĂŠnger recipients i "Sending..." limbo
! Lav ALDRIG .filter({ id }) re-fetch per recipient — forĂ„rsager Base44 rate limit (429)


=====================================================================
LAG 3: RATE LIMITING OG CRON INTERVAL
=====================================================================

Konfiguration:
  maxEmailsPerRun = 50
  batchSize       = 10
  delayMsBetween  = 400ms
  cronInterval    = 2 minutter

Beregning:
  50 emails × ~400ms = ~20 sekunder per kald
  Kald tager 20 sek, interval er 120 sek → 100 sek buffer → ingen overlap
  ~150 emails/minut (Brevo grĂŠnse: ~300/min)
  ~1.500 emails/time, ~36.000 emails/dag

Rate limit hÄndtering:
  let rateLimitBackoff = 1;

  if (brevoResponse.status === 429) {
    rateLimitBackoff = Math.min(rateLimitBackoff * 2, 10);
    await new Promise(r => setTimeout(r, 2000 * rateLimitBackoff));
    throw new Error('Rate limit');
  }

  rateLimitBackoff = 1; // nulstil efter success
  await new Promise(r => setTimeout(r, 400 * rateLimitBackoff)); // delay mellem batches

FALDGRUBER:
! SĂŠt ALDRIG cron til under 1 minuts interval
! Base44 functions timeout efter ~30 sekunder


=====================================================================
FULD KODE TEMPLATE — resumeBroadcastCampaigns
=====================================================================

import { createClientFromRequest } from 'npm:@base44/sdk@0.8.23';

Deno.serve(async (req) => {
  try {
    const base44 = createClientFromRequest(req);
    const BREVO_API_KEY = Deno.env.get('BREVO_API_KEY');

    const campaigns = await base44.asServiceRole.entities.BroadcastCampaign.filter({ status: 'sending' });
    if (campaigns.length === 0) return Response.json({ message: 'No campaigns to process' });

    for (const campaign of campaigns) {

      // 1. Hent alle unsent recipients (pagineret)
      const allRecipients = [];
      let skip = 0;
      while (true) {
        const batch = await base44.asServiceRole.entities.BroadcastRecipient.filter(
          { campaign_id: campaign.id, will_receive: true, sent: false },
          '-created_date', 500, skip
        );
        if (!batch || batch.length === 0) break;
        allRecipients.push(...batch);
        if (batch.length < 500) break;
        skip += 500;
      }

      // 2. In-memory deduplication (LAG 1)
      const seenEmails = new Set();
      const recipients = [], duplicates = [];
      for (const r of allRecipients) {
        const key = (r.email_normalized || r.email || '').toLowerCase().trim();
        if (!key || seenEmails.has(key)) { duplicates.push(r); continue; }
        seenEmails.add(key);
        recipients.push(r);
      }
      if (duplicates.length > 0) {
        await Promise.all(duplicates.map(r =>
          base44.asServiceRole.entities.BroadcastRecipient.update(r.id, {
            sent: true, sent_at: new Date().toISOString(), error_message: 'Skipped: duplicate'
          })
        ));
      }

      // 3. BegrĂŠns til max 50 per kald (LAG 3)
      const toSend = recipients.slice(0, 50);
      let sentCount = 0, failedCount = 0, rateLimitBackoff = 1;

      for (let i = 0; i < toSend.length; i += 10) {
        const batch = toSend.slice(i, i + 10);

        for (const recipient of batch) {

          // 4. Claim-before-send (LAG 2) — INGEN re-fetch (undgĂ„r Base44 rate limit)
          await base44.asServiceRole.entities.BroadcastRecipient.update(recipient.id, {
            sent: true, sent_at: new Date().toISOString(), error_message: 'Sending...'
          });

          try {
            const htmlContent = campaign.html_content
              .replace(/{{firstName}}/g, recipient.first_name || '')
              .replace(/{{email}}/g, recipient.email || '');

            const brevoRes = await fetch('https://api.brevo.com/v3/smtp/email', {
              method: 'POST',
              headers: { 'Content-Type': 'application/json', 'api-key': BREVO_API_KEY },
              body: JSON.stringify({
                sender: { name: campaign.from_name, email: campaign.from_email },
                to: [{ email: recipient.email }],
                subject: campaign.subject,
                htmlContent
              })
            });

            if (brevoRes.status === 429) {
              rateLimitBackoff = Math.min(rateLimitBackoff * 2, 10);
              await new Promise(r => setTimeout(r, 2000 * rateLimitBackoff));
              throw new Error('Rate limit');
            }
            if (!brevoRes.ok) throw new Error('Brevo: ' + brevoRes.status);

            await base44.asServiceRole.entities.BroadcastRecipient.update(recipient.id, {
              sent: true, failed: false, error_message: null
            });
            sentCount++;
            rateLimitBackoff = 1;

          } catch (error) {
            await base44.asServiceRole.entities.BroadcastRecipient.update(recipient.id, {
              sent: false, failed: true, error_message: error.message
            });
            failedCount++;
          }
        }

        if (i + 10 < toSend.length) {
          await new Promise(r => setTimeout(r, 400 * rateLimitBackoff));
        }
      }

      // 5. Tjek om kampagnen er fĂŠrdig
      const remaining = await base44.asServiceRole.entities.BroadcastRecipient.filter(
        { campaign_id: campaign.id, will_receive: true, sent: false }, '-created_date', 1
      );
      await base44.asServiceRole.entities.BroadcastCampaign.update(campaign.id, {
        status: remaining.length === 0 ? 'sent' : 'sending',
        sent_count: (campaign.sent_count || 0) + sentCount,
        failed_count: (campaign.failed_count || 0) + failedCount,
        ...(remaining.length === 0 ? { sent_at: new Date().toISOString() } : {})
      });
    }

    return Response.json({ success: true });

  } catch (error) {
    return Response.json({ error: error.message }, { status: 500 });
  }
});


=====================================================================
TJEKLISTE
=====================================================================

DATABASE / ENTITETER:
  [ ] BroadcastRecipient har email_normalized felt (lowercase/trimmed)
  [ ] BroadcastRecipient har sent (boolean), failed (boolean), error_message (string)
  [ ] BroadcastCampaign har status enum: draft, sending, sent
  [ ] BroadcastCampaign har sent_count, failed_count, will_receive_count felter

BACKEND FUNKTION:
  [ ] Pagineret hentning af recipients (while-loop, 500 per side)
  [ ] In-memory deduplication pÄ email_normalized FØR afsendelse
  [ ] Duplikater markeres sent: true med error_message: "Skipped: duplicate"
  [ ] Max 50 recipients per kald
  [ ] Claim-before-send: sÊt sent: true med "Sending..." FØR Brevo-kald
  [ ] INGEN re-fetch (.filter({ id })) per recipient — forĂ„rsager Base44 rate limit!
  [ ] SUCCESS path: opdater med failed: false, error_message: null
  [ ] FEJL path: revert til sent: false, failed: true, error_message
  [ ] 429 rate limit hÄndtering med eksponentiel backoff
  [ ] 400ms delay mellem batches af 10
  [ ] Kampagne markeres status: sent nÄr alle recipients er behandlet

CRON JOB:
  [ ] Interval: 2 minutter (IKKE under 1 minut)
  [ ] Cron kalder KUN resumeBroadcastCampaigns
  [ ] Ingen andre aktive cron jobs med legacy email-sending

DEPREKEREDE FUNKTIONER (skal deaktiveres):
  [ ] processScheduledEmails — returner 200 no-op
  [ ] sendEmailBatch — returner 410 Gone
  [ ] sendEmailBatchV3 — returner 410 Gone


=====================================================================
FEJLSCENARIER OG LØSNINGER
=====================================================================

SCENARIO 1: Emails sendt dobbelt
  Symptom: Modtagere klager over at have modtaget samme email 2 gange
  Årsag:   Manglende claim-before-send ELLER cron interval for kort
  LĂžsning: Implementer claim-before-send (uden re-fetch) + sĂŠt cron til 2 minutter

SCENARIO 2: Kampagne hÊnger pÄ status "sending" / sender ikke
  Symptom: Kampagne viser "sending" men ingen emails sendes, sent_count stiger ikke
  Årsag A: Recipients er i limbo — sent: true med "Sending..." placeholder
  Årsag B: Base44 rate limit (429) — for mange DB-kald per run (fx re-fetch per recipient)
  LĂžsning A: KĂžr recovery script:
  LĂžsning B: Fjern re-fetch (.filter({ id })) per recipient fra koden

  const stuck = await base44.asServiceRole.entities.BroadcastRecipient.filter({
    campaign_id: 'DIN_CAMPAIGN_ID',
    error_message: 'Sending...'
  });
  await Promise.all(stuck.map(r =>
    base44.asServiceRole.entities.BroadcastRecipient.update(r.id, {
      sent: false, failed: false, error_message: null
    })
  ));

SCENARIO 3: Mange fejlede recipients
  Symptom: failed_count stiger hurtigt
  Årsag:   Brevo rate limit (429) eller ugyldige emails
  LÞsning: Tjek error_message pÄ recipients. SÊnk maxEmailsPerRun til 30.

SCENARIO 4: Duplikater i database
  Symptom: Samme email har to BroadcastRecipient records
  Årsag:   generateBroadcastRecipients er kþrt to gange
  LÞsning: In-memory deduplication (LAG 1) hÄndterer dette automatisk.
           De markeres med error_message: "Skipped: duplicate"
Finvo

A simpler way to explore borrowing in South Africa. Compare options from NCR-registered lenders with our free, independent service.

Contact

Finvo is operated by Nexa Finance A/S

Company details and terms

info@finvo.co.za

Finvo is not a credit provider. We are an independent loan comparison service and only work with credit providers registered with the National Credit Regulator (NCR). When you apply through Finvo, your request is passed on to one or more lenders, who make their own credit decision. Our service is free – we may receive a commission from our partners when a loan is paid out, which never affects your interest rate or fees.

Representative example: A loan of R30,000 over 36 months at an illustrative annual interest rate of 28.25% gives an estimated monthly repayment of R1,364, including the initiation fee and monthly service fee. Total repayment: R49,107. Repayment terms range from 6 to 72 months.

© 2026 Finvo. All rights reserved.