Vendor configuration overview

How vendor-driven retail invoice fetching works in PayPulse Cloud.

Retail invoice fetching in PayPulse Cloud is driven entirely by configuration stored in the VendorConfig DynamoDB table. Adding a new vendor — or disabling an existing one — requires no Lambda code changes.

The VendorConfig table

The VendorConfig table is defined in aws-infra-terraform/dynamodb.tf with vendor_id as its partition key and PAY_PER_REQUEST billing.

Attribute	DynamoDB type	Description
vendor_id	String (PK)	Unique identifier for the vendor (e.g. dominos, zalando). Used as the S3 path segment and DynamoDB key.
vendor_name	String	Human-readable display name (e.g. Dominos, Jack & Jones).
invoice_category	String	Always retail for vendors in this table.
invoice_sub_type	String	One of the 8 retail categories (see below). Determines the S3 sub-path.
default_email_patterns	String Set	One or more sender email addresses to match (e.g. [e-mail]).
default_subject_keywords	String Set	One or more subject-line keywords to filter emails by (e.g. Tack for din beställning).
parser_type	String	Format of the invoice content — currently html for all vendors.
active	Boolean	When true, this vendor is included in every fetch run. Set to false to disable without deleting the record.
supports_pdf	Boolean	Whether this vendor sends PDF attachments.
supports_html	Boolean	Whether this vendor sends HTML email bodies.
logo_url	String	URL to the vendor logo asset in S3. May be empty.
created_at	String	ISO 8601 creation timestamp.
updated_at	String	ISO 8601 last-update timestamp.

Example record

The Zalando vendor config from vendor_configs/clothing/zalando.json:

{
  "vendor_id": {"S": "zalando"},
  "vendor_name": {"S": "Zalando"},
  "invoice_category": {"S": "retail"},
  "invoice_sub_type": {"S": "clothing"},
  "default_email_patterns": {"SS": [
    "[e-mail]"
  ]},
  "default_subject_keywords": {"SS": [
    "Thanks for your order"
  ]},
  "parser_type": {"S": "html"},
  "active": {"BOOL": true},
  "supports_pdf": {"BOOL": false},
  "supports_html": {"BOOL": true},
  "logo_url": {"S": "https://paypulse-vendor-logos.s3.eu-west-1.amazonaws.com/svgs/zalando.svg"},
  "created_at": {"S": "2025-10-06T10:30:00Z"},
  "updated_at": {"S": "2025-10-06T10:30:00Z"}
}

The 8 retail categories

Every vendor maps to one invoice_sub_type. This value determines the S3 path segment under invoices/{user_id}/retail/ where the HTML email is stored, and which DynamoDB detail table the parsed invoice is written to.

food-delivery

Restaurant and food delivery orders (e.g. Dominos, Foodora).

clothing

Fashion and apparel purchases (e.g. Zalando, Jack & Jones, Fotproffsen).

technology

Electronics and tech products.

subscriptions

Streaming services, SaaS receipts, memberships (e.g. Anthropic, Mevlana Moské).

grocery

Supermarket and grocery store purchases.

utility

Electricity, water, internet, and other utility bills.

travel

Transportation invoices — flights, trains, buses.

miscellaneous

Other retail purchases that don’t fit a named category.

Each sub-type has a dedicated DynamoDB detail table (FoodDeliveryInvoices, ClothingInvoices, TechnologyInvoices, etc.) in addition to the shared RetailInvoices base table.

How fetch_retail_invoices reads vendor configs

The fetch_retail_invoices Lambda (lambdas/invoices/fetch_retail_invoices/lambda_function.py) runs the following flow on every POST /v1/invoices/retail/ingest call:

1. Load active vendors

   The Lambda calls get_active_vendors(vendor_config_table) to scan the VendorConfig table and return only records where active = true. An optional vendor_category filter in the request body narrows this to a single sub-type.

2. Determine the date range

   determine_search_date_range() checks the last_retail_invoice_fetch field in the Users table for the authenticated user.

   • First fetch: searches the last 30 days.
   • Subsequent fetches: searches from last_retail_invoice_fetch to now.
   • Custom range: if start_date and end_date are provided in the request body (format YYYY-MM-DD), those take precedence.
3. Build the Gmail search query

   For each vendor, build_gmail_query() combines:

   • Sender filter: from:([e-mail]) — one OR-joined entry per address in default_email_patterns.
   • Subject filter: subject:(Tack for din beställning) — one OR-joined entry per keyword in default_subject_keywords.
   • Date filter: after:YYYY/MM/DD before:YYYY/MM/DD.

   If default_email_patterns is empty for a vendor, that vendor is skipped with a warning.

4. Search Gmail and collect emails

   The Lambda calls the Gmail API messages.list endpoint with the generated query, capped at 100 results per vendor to stay within the Lambda timeout. For each matching message, it fetches the full email content.

5. Duplicate detection

   Before uploading, the Lambda generates a deterministic S3 key (generate_retail_invoice_s3_key) from user_id, vendor_id, sub_type, email date, and Gmail message ID. It then calls s3_file_exists() — if the object already exists, the email is skipped. This prevents re-processing the same invoice on repeated fetch calls.

6. Upload to S3

   New emails are extracted as HTML (extract_html_from_email) and uploaded to:

   {bucket}/invoices/{user_id}/retail/{sub_type}/{vendor_id}_{date}_{message_id_hash}.html
   
7. Update fetch timestamp

   After processing all vendors, update_last_retail_invoice_fetch() writes the current UTC timestamp back to the Users table so the next call fetches only new emails.

A 100 ms sleep between individual emails and a 200 ms sleep between vendors prevents Gmail API rate-limit errors.

Incremental fetching

The last_retail_invoice_fetch field in the Users table is an ISO 8601 timestamp (e.g. 2025-10-06T10:30:00Z). On each successful run it is updated to the current UTC time, so subsequent calls only search the window between the last fetch and now.

If the stored timestamp cannot be parsed, the Lambda falls back to searching the last 30 days and logs a warning.

Duplicate detection

Duplicate detection happens at the S3 layer before any upload. The S3 key encodes the Gmail message ID, so the same email always produces the same key. If s3:HeadObject confirms the object exists, the email is skipped without triggering the downstream parser. This means calling POST /v1/invoices/retail/ingest multiple times is safe — only genuinely new emails are processed.

Currently configured vendors

Vendor	Sub-type	Sender
Dominos	food-delivery	[e-mail]
Foodora	food-delivery	[e-mail]
Zalando	clothing	[e-mail]
Fotproffsen	clothing	[e-mail]
Jack & Jones	clothing	[e-mail]
Anthropic	subscriptions	[e-mail]
Mevlana Moské	subscriptions	[e-mail]