Skip to content

Flow: Settings Sync (DynamoDB → Cloudflare KV)

How index settings changes propagate from DynamoDB through the settings exporter Lambda to Cloudflare KV, where the search proxy reads them.

Request Path

graph TD
    A["Admin API (or admin Lambda)"]

    subgraph ddb["DynamoDB: EcomIndexSettingsTable"]
        subgraph stream["DynamoDB Stream (NEW_IMAGE, 10s batch window)"]
            subgraph lambda["Lambda: {env}-EcomSettingsExporterLambda"]
                B["Build KvSettings from stream record"]
                C["Fetch API key from API keys service"]
                D["Cloudflare KV API: bulk PUT (two keys per index)"]
            end
        end
    end

    E["Cloudflare KV (search_proxy reads on next request)"]

    A --> B
    B --> C
    C --> D
    D --> E

Step-by-Step

1. Settings Write to DynamoDB

Where: Various sources write to {env}-EcomIndexSettingsTable:

  • Admin Lambda (components/admin_lambda/) — updates from admin dashboard
  • Shopify Admin Server — initial setup during shop onboarding

Table schema: pk={system_account_id}, sk=INDEX#{index_name}. Contains: index endpoint, search config, collections config, feature flags, agentic config, pixel_id, aliases, query override sharding metadata.

Inspect: Check settings record — see DynamoDB and Ecommerce.

aws dynamodb get-item --table-name {env}-EcomIndexSettingsTable \
  --key '{"pk": {"S": "the-account-id"}, "sk": {"S": "INDEX#the-index-name"}}'

2. DynamoDB Stream Trigger

Where: infra/ecom/stacks/ecom_stack.py

  • Stream view type: NEW_IMAGE (complete new record in each event)
  • Batching window: 10 seconds
  • Starting position: LATEST (no backfill on deploy)

The stream fires for INSERT, MODIFY, and REMOVE events.

3. Settings Exporter Lambda

Where: components/ecom_settings_exporter/ecom_settings_exporter/lambda_function.py

Two invocation modes:

  • Stream event: Processes DDB stream records (normal path)
  • Manual invocation: {"system_account_id": "...", "index_name": "..."} for recovery/re-export

For each stream record:

  1. Deserialize DynamoDB format → Python dict
  2. Skip default config records (sk == DEFAULT_CONFIGS_SK)
  3. Fetch API key from API keys service (for the account's cell)
  4. Build KvSettings Pydantic model: marqo_api_key, index URL, search config, feature flags, aliases, etc.
  5. Convert legacy read_alias → new index_aliases format if needed

Inspect: Check exporter Lambda logs — see Lambda.

aws logs tail /aws/lambda/{env}-EcomSettingsExporterLambda --since 15m

4. Cloudflare KV Write

Where: components/ecom_settings_exporter/ecom_settings_exporter/cloudflare_kv_store_client.py

Two KV keys written per index:

  • Primary: {shop_id} (most common lookup path, used by Shopify platform)
  • Alias: {system_account_id}-{index_name} (used by direct API via x-marqo-index-id header)

Uses Cloudflare REST API bulk PUT (up to 10,000 keys per request). Retries up to 3 times on 5xx errors.

KV namespace: Configured per environment in wrangler.toml (binding KV).

Inspect: Check KV — see Cloudflare Workers.

5. Search Proxy Reads Settings

Where: components/search_proxy/src/direct/platform.ts

On each request: env.KV.get(shopId) with configurable cache TTL. Parsed with Zod (MarqoSettings schema) — unknown fields allowed for forward compatibility.

Query Config Override Sync (Same Stream, Different SK Pattern)

Query-config overrides flow through the same EcomIndexSettingsTable stream, not a separate table:

  1. Written to {env}-EcomIndexSettingsTable with sk=INDEX#{index_name}#QUERY#{normalized_query} (same table as index settings, different sort key pattern)
  2. Same DDB Stream triggers the settings exporter Lambda
  3. Exporter detects #QUERY# in the sort key and calls _build_qcfg_kv_item()
  4. Writes KV keys: qcfg-{account}-{index}-{sha256(normalized_query)}
  5. Search proxy looks up via KV_QCFG binding with XOR filter for membership test

Note: EcomIndexQueryConfigsTable exists as a separate table but is not the source for stream-based KV export. The stream-based export path uses records in EcomIndexSettingsTable.

Inspect: Check KV_QCFG namespace (separate from main settings KV).

Merchandising Exporter (Separate Path)

Where: components/merchandising_exporter/src/lambda_function.py

  • Trigger: EventBridge schedule (every 5 minutes)
  • Scans {env}-MerchandisingTable for recently updated indexes
  • Async Lambda invocation per index
  • Writes to separate Cloudflare KV namespace

Inspect: Check merchandising exporter Lambda — see Lambda.

aws logs tail /aws/lambda/MerchandisingExporterLambda-{env} --since 15m

Alias Deletion Safety

On REMOVE events, the exporter only deletes KV entries if no active aliases point to the removed index. This prevents orphaning live traffic routes.

What to Look For

Symptom Where to Check
Settings not reaching search proxy Exporter Lambda logs. KV write errors? API key fetch failed?
Stale settings after update DDB Stream delay (up to 10s batch). KV cache TTL in search proxy.
Missing API key in KV API keys service reachable? Cell gateway up?
Exporter Lambda errors CloudFlare API token valid? Check Secrets Manager ({env}/CloudFlareApiToken).
Query overrides not working QCFG records in EcomIndexSettingsTable (sk contains #QUERY#)? XOR filter rebuilding? Check qcfg_* logs in search proxy.
Merchandising not syncing EventBridge schedule running? Check merchandising exporter Lambda.
Settings correct in DDB but wrong in KV Manual re-export: invoke Lambda with {"system_account_id": "...", "index_name": "..."}. Add "include_query_configs": true to also re-export query configs.

Monitoring

Where: components/ecom_monitoring_service/

The monitoring service Lambda (runs every 10 minutes) verifies:

  • All indexes in DDB settings table actually exist in Marqo
  • CloudWatch alarms exist for active indexes
  • Reports orphaned settings and missing alarms

Inspect: Check monitoring Lambda — see Lambda.