Skip to content

Reindexing

Background

Some index settings are fixed at index creation time, and changing them requires recreating the index. Notably this includes the model, which is prone to change over an index’s lifetime with iterative fine-tuning improvements.

However recreating an index is messy: the user must send all the existing docs again, monitor and deal with any failures, and stop/start data pipelines syncing into Marqo. This is unacceptable; we need to be able to upgrade a model and reindex the docs in the index relatively transparently so the customer is not interrupted.

Summary

Upgrading an index’s model and reindexing its docs is like a blue-green deployment. The current, working index is basically left alone while a new one is spun up next to it with the new model and populated with the existing docs. The trick is that while the new index is being set up, the stream of incoming write requests to the existing index’s indexing queue is replicated to the new index’s own indexing queue so it doesn’t fall behind.

Process

  • Decide on the index settings you want the new index to have
    • create_index_config must be defined create_index_config understand whether to set textQueryPrefix on create_index_config
    • if so, set on DEFAULT_CONFIG record before creating
  • index alias so the same ecom API URL can write to the new index
    • maybe optional if customer is happy to hit a new URL
  • create new index with new queue with existing config on prod-au-index (crucially using the right collapseFields on parentProductId)

  • update syncservice to write to two queues

  • runbook:

    • update INDEX#DEFAULT_CONFIGS with whatever create_index_config you want for the new index (e.g. model, collapseFields, textQueryPrefix, etc.)

      • Ideally also define the add_docs_config, search_config, and collections_config.default

      • Everything the same as the current config here , EXCEPT:

        • Model name (new model)
        • change the weights of the multimodal field, so change variantTitle to be slug e.g. old version:

        image.png

    • create new index with new model

    • update existing index with a new WriteAlias in add_docs_config. This will instruct the ecom lambda to put a job and SQS message for each alias.

    • run Richie pipeline targeting new index

    • OPTIONAL: once reindexed update search alias (not implemented)
    • hook up ETL pipeline
    • Migrate merchandising rules, query configs, redirects, etc.

Tasks

  • Prepare index settings @Nick
  • Create new index in Kogan’s account with a similar but different name @Ruchira Jayasekara
    • Could be: prod-au-index-2 or use the date convention, prod-au-index-20251121
  • Build write request fanout in SyncService @Oliver Lade
  • Write doc on ecom throughput levers @Oliver Lade
  • Add new index’s queue URL to existing index’s settings when list of queues is supported @Oliver Lade and validate it receives messages
  • (lower priority, may not be needed) Add query prefix feature to search_proxy @Oliver Lade
    • And update settings exporter to sync it from DDB