The LiteSpeed Cache crawler visits the URLs in a site's XML sitemap on a schedule so LiteSpeed builds a cached copy of each page before a real visitor asks for it. Without it, the first visitor after a purge or a cache expiry gets the slow, uncached page. To use the LiteSpeed Cache crawler, enter a sitemap URL in LiteSpeed Cache > Crawler > Settings and turn it on. This guide covers that, plus how to tune it and how to tell whether it's working.
The steps match LiteSpeed Cache 7.x. Version 7.0 removed several crawler settings, so older guides you find online may show options you won't see. See Older guides show settings you can't find below.
LiteSpeed caches a page the first time someone requests it. Every page is slow once: after a purge, after the cache expires, and after you publish changes. On a busy site, visitors absorb that cost and you never notice. On a site with hundreds of pages and modest traffic, most visits to deep pages can land on an uncached copy.
The crawler absorbs that cost instead. It reads the sitemap, requests each URL the way a visitor would, and LiteSpeed caches the response. It runs from WP-Cron, in short batches, and it backs off when the server is busy.
It does not make an uncacheable page cacheable. Cart, checkout, account pages and anything else that sends a no-cache header are skipped and listed on the crawler's Blocklist.
Turn it on when:
Leave it off when:
The crawler uses the same CPU and PHP workers as the site. It is a trade: server work now, so visitors don't wait later.
LiteSpeed Enterprise lets the host decide whether WordPress sites on a server may run the crawler. On WP Aligned hosting it's enabled on every server, so you only need to set it up in the plugin.
If the Summary tab at LiteSpeed Cache > Crawler ever shows this warning on one of our servers, open a ticket and include the site's domain:
The crawler feature is not enabled on the LiteSpeed server. Please consult your server admin or hosting provider.
The crawler needs an XML sitemap or sitemap index. Open the one your site already has in a browser to confirm it loads:
| Sitemap source | Usual URL |
|---|---|
| WordPress core (no SEO plugin) | https://example.com/wp-sitemap.xml |
| Yoast SEO | https://example.com/sitemap_index.xml |
| Rank Math | https://example.com/sitemap_index.xml |
| All in One SEO | https://example.com/sitemap.xml |
A sitemap index is fine. The crawler follows it into the child sitemaps.
Custom Sitemap is required. If it's empty, the Summary tab says You need to set the Custom Sitemap in Settings first before using the crawler and the Manually run button stays disabled.
The other settings on that tab:
1. The crawler uses up to 3 threads, drops threads when load goes above this limit, and stops the run if it can't get under the limit with one thread.Don't raise Server Load Limit to make the crawler go faster. That limit is what keeps the crawler from slowing down the site it's trying to speed up, and the site shares its server with other sites.
A manual run uses the same batch limits as the scheduled one, so on a large site the first full pass takes several runs. After that, WP-Cron starts a new batch every 10 minutes until the sitemap is done, then waits for the Crawl Interval.
The crawler is scheduled by WP-Cron, and WP-Cron only runs when someone loads a page. Our servers don't run a cron job for WordPress on your behalf, so on a quiet site the crawler can sit idle for hours, which is exactly when it's supposed to be warming pages. We recommend you set up a server cron job for every site that uses the crawler. It takes about five minutes in cPanel.
LiteSpeed Cache > Crawler > Map lists every URL the crawler read from the sitemap. The colored badge in the Crawler Status column shows the result of the last visit:
The crawler works in batches, so on a first run it's normal to see gray rows. They fill in over the next few runs.
If you changed the sitemap, click Refresh Crawler Map so the crawler reads it again.
Pick a URL the Map tab shows as crawled. Purge nothing, stay logged out, and request it:
curl -sI https://example.com/some-page/ | grep -i x-litespeed-cache
x-litespeed-cache: hit on the very first request you make means the crawler warmed it. miss means it wasn't cached yet, or the cached copy was purged since the crawler's visit.
On a big site, a full crawl can take days and spend most of its time on pages nobody visits. Point the crawler at a smaller sitemap instead:
https://example.com/page-sitemap.xml and https://example.com/post-sitemap.xml, and skip tag, author and attachment sitemaps.https://example.com/wp-sitemap-posts-page-1.xml.Enter each sitemap on its own line in Custom Sitemap, save, then click Refresh Crawler Map on the Map tab.
Every extra variation multiplies the work. The crawler builds one crawler per combination of guest or role, mobile or desktop (when mobile cache is on), next-gen image support (when WebP or AVIF replacement is on), and each cookie value. Each crawler walks the whole sitemap. Add variations only when real visitors need them.
Role Simulation crawls as a logged-in user, which only helps if you cache pages for logged-in users (private cache with ESI). It needs two things first:
Then enter a user ID in Role Simulation. Use a dedicated low-privilege account with the role you want to warm, never an administrator.
Cookie Simulation crawls once per cookie value, for sites that vary pages by a cookie such as a currency or language switcher. Enter the cookie name and its values, one per line. Use _null as a value to also crawl the version where the cookie isn't set.
Either Custom Sitemap is empty, or the server-level warning is showing (see The crawler is enabled on WP Aligned servers above).
The crawler couldn't read any URLs from the sitemap. Check that:
Then click Refresh Crawler Map.
Look at Ended reason on the Summary tab. If it says stopped_highload, the crawler backed off because the server was busy. That is the load limit working. It will try again on the next run.
If the scheduled runs never start, check WP-Cron. The crawler is a WP-Cron job, and WP-Cron only fires when the site gets a page load, unless a server cron job calls it. If DISABLE_WP_CRON is set to true in wp-config.php and no server cron job is set up, the crawler never runs on its own. To fix both cases, set up a server cron job.
Pages are blocklisted when they send a no-cache header or no LiteSpeed cache header at all. Some of that is correct (cart, checkout, account). If normal content pages are blocklisted:
curl command above. If it never shows hit, the page isn't cacheable, and the crawler can't change that. Look for an exclusion in LiteSpeed Cache > Cache > Excludes, or a plugin that sends no-cache.LiteSpeed Cache 7.0 removed Delay, Run Duration, Interval Between Runs, Threads, Timeout and Drop Domain from Sitemap from the dashboard. Everything left is on one Settings tab under LiteSpeed Cache > Crawler. The removed values are now fixed in the plugin. If a guide tells you to raise threads or shorten the delay, skip it: those knobs aren't in the dashboard anymore, and on shared servers the load limit is the right control.
Open a ticket if:
The crawler feature is not enabled on the LiteSpeed server.curl never shows x-litespeed-cache: hit on pages the Map tab says it crawled.Include the site's domain and a screenshot of the Crawler Summary tab. See Need Help? for how to reach us.
No. It only fills it. To clear the cache, see How to Clear LiteSpeed Cache on WP Aligned Hosting.
Not directly. It requests pages, and if Redis is on, those requests fill it as a side effect. Redis is off by default on our hosting. See Redis Object Caching.
Server-side tools that read access logs will see the crawler's requests. Its user agent contains lscache_walker or lscache_runner, so you can filter it out. JavaScript-based analytics such as Google Analytics don't record it, because the crawler doesn't run JavaScript.
It depends on the number of URLs, the number of crawler variations, and how busy the server is. The Summary tab shows Last complete run time for all crawlers once one full pass has finished.