| page_type | languages | name | description | products | urlFragment | |||
|---|---|---|---|---|---|---|---|---|
sample |
|
Optimize indexing with the push API |
In Azure AI Search, you can push data from any data source to an index. In this C# sample, learn how to index more efficiently. |
|
optimize-data-indexing |
Azure AI Search supports two basic approaches for importing data into a search index: pushing your data into the index programmatically, or pointing an Azure AI Search indexer at a supported data source to pull in the data.
As data volumes grow or processing needs change, you might find that simple or default indexing strategies are no longer practical. This sample demonstrates how to efficiently index data using the push model by batching requests and leveraging an exponential backoff retry strategy.
This .NET Core console app builds off of the code used in the Quickstart and uses the Azure AI Search .NET SDK to create an index and efficiently load it with documents using the push model.
The app shows how to:
- Test different batch sizes to understand the most efficient batch size for indexing your data
- Efficiently index data by:
- Pushing data to the index in batches
- Sending multiple batches concurrently
- Implementing an exponential backoff retry strategy
The index is modeled on a subset of the Hotels dataset, reduced for readability and comprehension. Index definition and documents are included in the code.
Note
Network transfer speeds can be a limiting factor when indexing data. You might get a better sense of indexing efficiency if your Visual Studio client is on an Azure virtual machine in the cloud in the same location as Azure AI Search. The Data Science VM is a good choice because Visual Studio is preinstalled.
- Clone or download this sample repository.
- Extract contents if the download is a zip file. Make sure the files are read-write.
This sample is available in two versions.
Use v11 version and the Azure.Search.Documents client library. The v10 version is now obsolete and will be archived.
-
Open the OptimizeDataIndexing.sln project in Visual Studio.
-
Update appsetting.json to use your search service name and admin api-key. The admin key is necessary for creating objects and loading data.
-
Press F5 to build and run the project. On the first run, the sample prints batch size metrics to the console.
-
Uncomment lines 41 through 49 and rerun and the program. On this run, the sample generates and sends batches of documents, up to 100,000 if you run the code without changing the parameters. This simulation can help you understand how quickly indexing occurs for a given batch size.
You can learn more about Azure AI Search on the official documentation site.
The documentation provides additional guidance on indexing large data sets.