Batch → Limit Concurrency → Retry → Pause → Repeat

• Batch size: 25–100

• Concurrency: 3–5 (start here)

• Catch all failed items and retry on failed

• Delay: 200–500ms between batches

Step 1: Install and set up p-limit

npm install p-limit

import pLimit from "p-limit";

const CONCURRENCY = 5;
const limit = pLimit(CONCURRENCY);

const CONCURRENCY = 5; Defines that no more than 5 async tasks should run simultaneously.
const limit = pLimit(CONCURRENCY); Creates a function (limit) that wraps async operations and ensures that at most 5 are running at any given time.

Step 2: Retry with exponential backoff

const sleep = (ms: number) => new Promise(res => setTimeout(res, ms));

  const retryWithBackoff = async (
  fn: () => Promise<any>,
  retries = 3,
  delay = 500
) => {
  try {
    return await fn();
  } catch (err: any) {
    const status = err?.response?.status;

    if (retries > 0 && (status === 429 || status >= 500)) {
      await sleep(delay);
      return retryWithBackoff(fn, retries - 1, delay * 2);
    }

    throw err;
  }
}; 

It tries to execute an async function (fn). If the operation fails because the server is overloaded or temporarily unavailable, it waits for some time and tries again. Each retry waits longer than the previous one. It handles Rate limiting (HTTP 429), Temporary server failures (HTTP 500, 502, 503, 504, etc.) and Unstable network conditions.

Step 3: Wrap your Sitecore call

const syncToSitecore = async (item: any) => {
  return retryWithBackoff(async () => {
    const res = await fetch("/api/sitecore-sync", {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
      },
      body: JSON.stringify(item),
    });

    if (!res.ok) {
      const error: any = new Error("Request failed");
      error.response = { status: res.status };
      throw error;
    }

    return res.json();
  });
};
 

Step 4: Add batching + p-limit

const BATCH_SIZE = 50;

const processItems = async (items: any[]) => {
  for (let i = 0; i < items.length; i += BATCH_SIZE) {
    const batch = items.slice(i, i + BATCH_SIZE);

    console.log(`Processing batch ${i / BATCH_SIZE + 1}`);

    const tasks = batch.map(item =>
      limit(() => syncToSitecore(item))
    );

    await Promise.all(tasks);

    await sleep(300); // small pause to avoid throttling
  }
};

This code processes a large list of items and sends them to Sitecore in batches, while also limiting concurrency and adding retries. It

• Splits the items into groups of 50.

• Processes one group at a time.

• Within each group, only 5 requests run simultaneously (because of your earlier p-limit(5)).

• Retries failed requests using retryWithBackoff().

• Waits 300 ms between batches.

Step 5: Add failure tracking (don’t skip this)

const failedItems: any[] = [];
const safeSync = async (item: any) => {
  try {
    return await syncToSitecore(item);
  } catch (err) {
    failedItems.push(item);
  }
};
Then use:
const tasks = batch.map(item =>
  limit(() => safeSync(item))
);

Optional: Adaptive throttling

let dynamicDelay = 300;

if (res.status === 429) {
  dynamicDelay += 200;
}
Then:
await sleep(dynamicDelay);

Where to run this code

Better options

Why

What NOT to do

await Promise.all(items.map(syncToSitecore));

This causes:

Tuning guidelines

Start conservative:

Then adjust:

Final Takeaway

In conclusion, synchronizing large datasets into Sitecore XM Cloud is most effective when controlled concurrency is combined with the platform’s built-in scalability and content delivery capabilities. While p-limit serves as the essential concurrency guardrail that prevents API overload, it becomes truly production-ready when paired with batching, retry mechanisms, and request pacing, helping mitigate issues such as rate limiting, dropped requests, and long-running process failures. Sitecore XM Cloud’s API-first architecture, scalable SaaS infrastructure, Experience Edge, and content management APIs work seamlessly alongside Next.js features such as Static Site Generation (SSG), Incremental Static Regeneration (ISR), intelligent caching, and On-Demand Revalidation to ensure content updates are processed efficiently and delivered quickly to end users. Together, these capabilities provide high success rates, predictable performance, and resilience under load, enabling organizations to reliably synchronize thousands of content items while maintaining fast, scalable, and always-fresh digital experiences.