Technical SEO Checklist 2026

Why technical SEO checklists need an update

Most technical SEO checklists still in circulation were written before AI crawlers became a meaningful share of traffic hitting the average site's server, and before Core Web Vitals' metric set changed. Two things have shifted enough to matter for 2026 specifically: Interaction to Next Paint (INP) replaced First Input Delay (FID) as the responsiveness metric Google actually measures, and a genuinely new category of bot traffic — AI training scrapers and AI retrieval agents — now hits sites of every size, not just enterprise-scale catalogs. This checklist covers both the fundamentals that haven't changed and the parts that have.

1. Crawlability and indexing — the foundation

Nothing else on this list matters if crawlers can't reach your pages in the first place. Before anything else:

• Confirm your content is accessible in raw HTML before JavaScript runs. Client-side-rendered content that only appears after JavaScript execution is a persistent risk — verify with a raw HTML fetch, not just a rendered browser view.
• Verify mobile crawling separately from desktop. Google's primary crawler is the Googlebot Smartphone user agent — hidden mobile navigation (a hamburger menu whose links aren't in the DOM until clicked) can conceal your site structure from the crawler that actually determines your indexing, even if desktop navigation looks fine.
• Cross-reference your XML sitemap against what's actually discovered. Pull your submitted sitemap URLs, your crawl data, and your Search Console-reported organic-traffic URLs, then combine and deduplicate all three. Any URL that shows up in analytics or the sitemap but not in your crawl data is functionally an orphan page — invisible to crawlers navigating purely through your internal link graph.
• Remember that robots.txt doesn't control indexing on its own. It manages crawl access; a page can still be indexed from external links even if you've disallowed crawling it. Use noindex directives specifically when the goal is keeping a page out of the index.

2. Core Web Vitals: the 2026 thresholds

INP is the metric most sites still have stale advice about — it replaced First Input Delay (FID) as Google's responsiveness metric, and it measures every interaction throughout a page visit rather than just the very first one, which makes it a meaningfully stricter bar. A page that scored well on FID because the first click was fast can still fail INP if scrolling, opening a menu, or submitting a form later in the visit feels sluggish. Use PageSpeed Insights and Search Console's Core Web Vitals report regularly, not as a one-time audit — regressions creep in with routine content and template changes.

3. Crawl budget vs. index budget — a distinction worth knowing

These are frequently conflated, but they're different constraints with different fixes:

• Crawl Budget is how many URLs a search engine bot is willing and able to crawl on your site in a given window — a bandwidth and server-capacity limitation.
• Index Budget is how many of your pages the search engine actually considers worth keeping in its index — a quality-driven limitation.

For most sites in 2026, Index Budget is the more common bottleneck, not Crawl Budget — bloat (thin tag archives, outdated or near-duplicate pages, low-value pagination) dilutes your domain-wide quality signal and can crowd out genuinely valuable pages from full indexing consideration. Pruning or consolidating low-value pages is a legitimate strategic move, not just housekeeping — it protects the index budget available for the pages that actually matter.

Even small sites can run into real Crawl Budget strain now, for a different reason: the sheer combined volume of AI bots (in addition to standard search crawlers) scraping the web means server load from crawler traffic alone has grown substantially, independent of your own traffic. A slow server response to an important indexing bot, caused by unrelated bot traffic competing for the same resources, is a genuinely new failure mode worth checking for.

4. AI crawler governance — the part most 2023-era checklists miss entirely

This is the single biggest addition to technical SEO since Core Web Vitals themselves. robots.txt used to be a simple allow/disallow list for a handful of known search crawlers. In 2026, it functions closer to a governance document — a deliberate, per-bot decision about which AI companies can access your content for training versus for real-time answer retrieval.

The distinction that matters:

• Training bots scrape your content to train future AI models. Blocking these does not affect your visibility in either traditional search results or AI-generated answers — it only controls whether your content becomes training data.
• Retrieval bots fetch your content in real time to answer a specific user question. Blocking these directly removes you from that engine's AI-generated answers and citations.

Bot behavior and user-agent strings change; verify the current list and each operator's documentation before finalizing your robots.txt, since new AI crawlers continue to appear.

A commonly recommended default approach: allow retrieval bots broadly so your content can be cited in real-time AI answers, while blocking training-specific bots if you want to opt your content out of model training — these are two separate decisions, not one setting. If your robots.txt has no AI-specific rules at all, most AI bots will simply follow your default User-agent: * directive, which means you may be allowing full access to every AI crawler, training and retrieval alike, without having made that choice deliberately. The overwhelming majority of sites currently fall into this "accidental default" category rather than having made an intentional call — checking and updating your robots.txt for this is one of the highest-leverage, lowest-effort items on this entire checklist.

5. Schema and structured data

Schema markup has moved from "nice-to-have polish" to a core part of the technical foundation, largely because AI retrieval systems read structured data directly rather than inferring meaning purely from prose. Priority order for most content sites:

1. Organization / Person — establishes entity identity, particularly via the sameAs property linking to external authoritative profiles
2. Article — authorship, publish and update dates, content type
3. FAQPage — standalone, directly citable Q&A pairs
4. HowTo — explicit step structure for procedural content
5. Review / AggregateRating — social proof, relevant for comparison and review content specifically

Validate schema regularly, not just at initial implementation — it breaks silently after template or CMS changes far more often than most site owners realize, and mismatched schema (describing content that isn't actually visible on the page) is treated as a negative quality signal rather than ignored.

6. Site architecture and internal linking

• Fix orphan pages — any page with no internal links pointing to it is effectively invisible to crawlers navigating your link graph alone, regardless of whether it's in your sitemap.
• Clean up redirect chains — each additional hop in a redirect chain adds latency and dilutes the crawl efficiency and link equity passed through it.
• Review crawl depth — pages buried many clicks deep from the homepage get crawled less frequently and carry less internal link equity; flatten hierarchy where it makes sense.
• Audit faceted navigation carefully if you run anything with filters (even a blog's tag/category system) — filter and sort-parameter combinations can create a combinatorial explosion of near-duplicate, low-value URLs that waste crawl resources without adding indexable value.

7. Log file analysis — verifying what actually happens

Third-party crawl simulators show you what a crawler might do. Server log files show what crawlers actually did, request by request — the only technique that gives ground truth rather than a simulation. For sites with server access, periodic log analysis is worth the setup:

• Confirm important bots are actually reaching the content you intend them to find — if you've deliberately allowed a retrieval bot like PerplexityBot, verify in your logs that it's genuinely crawling the pages you care about, not just hitting your homepage occasionally.
• Watch the split between indexation, training, and retrieval bot traffic — this three-way split barely existed as a concept a couple of years ago, and most sites still aren't separating it, which means most robots.txt decisions right now are being made without the data that would actually inform them.
• Look for wasted crawl activity — bots repeatedly hitting low-value parameter combinations, near-duplicate faceted URLs, or deprecated paths that should be pruned or blocked.

A practical audit sequence

Rather than tackling every item at once, this rough priority order reflects where problems compound the most if left unaddressed:

1. Crawlability and indexing fundamentals (nothing else matters until this is solid)
2. robots.txt — deliberate AI crawler decisions, not accidental defaults
3. Schema coverage and validation
4. Core Web Vitals (LCP, INP, CLS)
5. Internal linking and orphan-page cleanup
6. Index budget review (prune or consolidate low-value pages)
7. Log file analysis, if server access allows it

Common mistakes worth checking for specifically

• Treating robots.txt as "set once and forget" — the list of AI crawler user agents worth having an opinion on has grown substantially in a short period, and a file that hasn't been touched in a year is very likely making default decisions by accident rather than deliberate ones.
• Confirming crawlability with a rendered browser view instead of raw HTML — this misses exactly the client-side-rendering problem it's meant to catch, since the browser executes the JavaScript a crawler might not.
• Auditing desktop navigation and assuming mobile matches — since Google's primary crawler evaluates the mobile experience specifically, a clean desktop structure with hidden or JavaScript-dependent mobile navigation can silently undercut everything else on this checklist.
• Adding schema without a validation habit — implementation at launch is only the starting point; template changes, CMS updates, and content edits break schema silently far more often than most site owners check for.
• Pruning low-value pages without checking internal links first — removing or noindexing a page that's still linked from elsewhere on the site just creates a new crawl and user-experience problem in place of the old one.

Tools for running this checklist

A technical SEO audit doesn't require expensive tooling to get most of the way there, but a few categories of tools make each section of this checklist meaningfully faster:

For a deeper comparison of the SEO platforms that cover most of this checklist in one tool, see our Best SEO Tools Compared 2026 guide.

Frequently asked questions

Does blocking GPTBot hurt my Google rankings? No — GPTBot is OpenAI's training scraper, entirely separate from Googlebot. Blocking it affects only whether your content is used to train OpenAI's models, not your visibility in Google Search.

Should small sites care about crawl budget? More than in previous years — while crawl budget was historically an enterprise-scale concern, the combined volume of AI bots now scraping the web means even small sites can see real server strain, which can in turn slow down how quickly important indexing crawlers are served.

Is llms.txt required? Not yet a formal, universally-honored standard — treat it as a low-cost, forward-looking addition once your core technical foundation (crawlability, schema, Core Web Vitals) is solid, not as an urgent priority ahead of those fundamentals.

How often should I re-audit robots.txt for AI crawlers? At minimum whenever a new major AI crawler emerges, since the list of relevant user agents is still growing — a robots.txt file that hasn't been touched in a year or more is worth revisiting even without a specific trigger.

The bottom line

The core of technical SEO hasn't changed — crawlability, fast and stable page experience, clean architecture, and accurate structured data remain the foundation. What's genuinely new for 2026 is the AI crawler layer: robots.txt now requires deliberate, bot-by-bot decisions most sites are still making by accident, and getting that layer right is one of the highest-leverage, lowest-effort items available to any site owner right now.

Once your technical foundation is solid, see our AI Overviews SEO Guide for what to change in the content itself to actually earn AI citations on top of it.