The Boring Technical Stuff That Quietly Caps Your Rankings

Boring is good. Boring stays up.

I say that about servers, but it applies to technical SEO exactly as well. Nothing in this article will win you a ranking. Every single item on it can lose you one, and several can take your entire site out of Google in an afternoon.

That asymmetry is the whole point. Technical SEO is mostly one-time work with permanent consequences. Get it right once at build time and it costs nothing. Retrofit it later and it costs weeks. Ignore it and you'll spend years wondering why good content isn't performing.

Here's the list, in the order I audit a site I'm taking over.

Robots.txt, and the one line that deletes a business

Robots.txt is a plain text file at your domain root telling crawlers which paths they may fetch. Use it to block genuinely useless crawl space: internal search results, faceted filter combinations, cart and checkout paths, admin URLs. And to declare where your sitemap lives.

Two misunderstandings cause real damage. First, robots.txt blocks crawling, not indexing. A blocked URL can still appear in Google's index, listed by URL alone, picked up from external links. To keep a page out of the index you need a noindex meta tag, and you have to allow crawling so the tag can actually be seen. Blocking a page you've noindexed means Google never learns it's noindexed.

Second, never block your CSS and JavaScript files. Google has to render pages the way users see them.

And then there's Disallow: /. One line, and every search engine on earth stops looking. This ships to production from staging environments constantly. It's the first thing I check in any traffic-collapse diagnosis and the first thing to verify after any migration or launch.

The AI crawler decision you now have to make on purpose

This one is new enough that most businesses have never considered it, and it's a genuine fork in the road.

GPTBot from OpenAI, ClaudeBot from Anthropic, PerplexityBot, Google-Extended and CCBot all identify themselves and respect robots.txt. Blocking them keeps your content out of AI training data. It also removes you from the answer engines those systems power.

You cannot be cited by an engine that cannot read you. For any business that wants customers rather than a principled stand about training data, the correct default is to allow them.

There's a subtlety that catches people who do think about this: beyond the training crawlers, there are separate search-time fetchers, OAI-SearchBot and Claude-SearchBot, which power live browsing inside ChatGPT and Claude. Allowing the training bots while blocking the search fetchers, or the reverse, is a common misconfiguration that produces confusing results. Check both.

Sitemaps and crawl budget

An XML sitemap is a machine-readable list of your canonical, indexable URLs with last-modified dates. It's a discovery aid and a statement of intent, so include only pages you actually want indexed. No redirects, no noindexed pages, no duplicates. A dirty sitemap teaches Google to distrust your sitemap.

Keep the last-modified dates honest. Google has said it uses them when they're reliable, and they drive efficient re-crawling of updated content.

Submit in both Google Search Console and Bing Webmaster Tools. Bing matters more than its market share suggests, because its index feeds several AI answer engines, and most sites never verify it at all. Then watch the Pages report for the gap between discovered and indexed. That gap is your crawl health dashboard.

Crawl budget only becomes a real constraint past a few thousand URLs, which most service businesses never reach. If you do get there, the levers are: noindex the thin auto-generated pages, canonicalize parameter variants, block infinite crawl spaces, keep architecture flat, fix redirect chains, and keep the server fast. Google crawls fast hosts more.

Canonicals and index bloat

Indexation control is deciding, page by page, what belongs in the index. Left unmanaged, sites accumulate hundreds of thin, duplicate and parameter pages, and because Google's quality assessment operates partly at site level, junk in the index taxes your good pages too.

The canonical tag declares which URL is the master copy when content is duplicated: http and https variants, www and non-www, tracking parameter URLs, printer views, sorted category variants, syndicated copies. Every page should self-canonical by default as a defence against parameter duplication.

Canonicals are hints Google can override. Search Console's URL Inspection shows you the Google-selected canonical, and if it disagrees with yours, your signals conflict somewhere: internal links pointing at the non-canonical version, the sitemap listing the wrong one, or pages that aren't actually duplicates. Canonical, redirect, sitemap and internal links all have to agree on one version. Signal consistency is the entire game.

Use noindex for internal search results, thin tag and archive pages, thank-you pages and gated previews. Just remember the interlock: noindex requires crawlability to be seen.

Core Web Vitals

Speed affects rankings three ways. It's a direct if modest ranking input, it constrains how often you get crawled, and, by far the biggest effect, it drives user behaviour. Abandonment climbs steeply with load time, and users who bounce back to the results page feed negative engagement signals into ranking while destroying your conversion rate regardless.

Performance is rarely why you rank third instead of first. It's often why visitors don't become customers, which I argued at length in You Can Rank #1 and Still Lose the Client.

The three metrics, measured from real Chrome users at the 75th percentile:

• LCP (Largest Contentful Paint) at 2.5 seconds or less. How fast the main content appears. Cured by fast hosting, a CDN, an optimized hero image that's preloaded and never lazy-loaded, and minimal render-blocking CSS and JavaScript.
• INP (Interaction to Next Paint) at 200ms or less. How fast the page responds to taps and clicks. It replaced FID in 2024. Cured by less JavaScript, splitting long main-thread tasks, and taming third-party tags. Chat widgets and tag-manager sprawl are the usual culprits.
• CLS (Cumulative Layout Shift) at 0.1 or less. Visual stability. Cured by explicit dimensions on images, videos, embeds and ads, and reserving space for anything that loads late.

The optimization order that actually matters: good hosting first, because cheap shared hosting caps everything downstream. Then a CDN. Then image discipline, since images are usually most of your page weight. Then JavaScript austerity, and this is where WordPress and Shopify sites bleed out, because every plugin and app has a cost that nobody audits. Then critical CSS, caching, and self-hosted subset fonts.

Chase the field data, not the lab score. A 100 out of 100 Lighthouse score is a vanity milestone. Passing all three vitals in real-user data across your template types is the actual goal, and past that point further speed work buys conversions rather than rankings.

This is a large part of why I put client sites on hardware I control rather than shared hosting. You can optimize a page endlessly and still lose to a slow server.

Mobile-first is not a design preference

Google indexes and ranks the mobile version of your site. Full stop, universally, since 2023. If content, structured data or internal links exist on desktop but not mobile, they effectively do not exist.

What that requires: content parity, meaning the same text, links, schema and images on both, with no "condensed" mobile version that quietly drops sections. Responsive design on one URL. Tap targets around 48px so people don't mis-tap. Base font of 16px or more. No horizontal scrolling. And no intrusive interstitials, because a full-screen popup blocking content on arrival from search is an explicit negative signal. Cookie banners and legal notices are fine. The aggressive email-capture modal on first paint is not.

Content inside accordions and tabs on mobile is fully indexed and weighted, so collapsing things for mobile UX is safe.

Test on your actual phone, on cellular, not just in a desktop emulator. That's the empathy test automation can't run.

HTTPS, done correctly

Certificates are free. HTTPS is the floor, not a tactic. The SEO risk lives entirely in implementation.

Every HTTP URL should redirect to its HTTPS twin in a single hop. All four origin variants, http and https crossed with www and non-www, must resolve to one canonical origin. Zero mixed content. Internal links, canonicals, sitemaps and schema all referencing the HTTPS canonical. A sloppy HTTPS migration is a self-inflicted traffic drop. A clean one is invisible.

Broader security is SEO too. A hacked site collects injected spam pages, a "this site may be hacked" label in results, Safe Browsing blocklisting and manual actions. That's traffic obliteration taking months to recover from even after cleanup. Keep things updated, use 2FA, take backups, and watch Search Console's Security Issues report, because that's where Google tells you first. Boring maintenance is a ranking strategy.

JavaScript, and why your framework choice is an SEO decision

Crawlers read HTML first. If your HTML is an empty div and the content only exists after JavaScript executes, you're betting your visibility on a rendering queue that can lag the initial crawl by hours or days.

In order of preference: static site generation, server-side rendering, hybrid approaches, and dynamic rendering as a legacy patch. Client-side rendering alone, for content you want indexed, is malpractice in 2026.

Audit it with View Source, not the DevTools inspector. The inspector shows you the rendered result. View Source shows you what the crawler got. Common failures even on server-rendered sites: links built as click handlers instead of real anchor tags, since crawlers follow hrefs and don't click. Content that loads on scroll, which crawlers never trigger because they don't scroll. Meta tags and canonicals injected client-side. Hash-based routing, where everything after the # isn't a distinct URL at all.

The stakes went up with AI search. Many AI-engine crawlers don't render JavaScript at Google's level, or at all. A JavaScript-dependent site can be adequately visible to Google and completely invisible to the systems behind ChatGPT and Perplexity. Server-rendered HTML is the only safe substrate now, which is the practical reason my builds are Django rendering HTML on the server.

Redirects and migrations, where years of work die

Most catastrophic organic traffic losses are self-inflicted migration failures. This section is cheap insurance.

Use 301 for permanent moves; it passes essentially full equity. Use 302 only for genuinely temporary ones. Redirect to the most relevant live page, never bulk-redirect everything to the homepage, because Google treats that as a soft 404 and passes nothing. One hop only, since chains leak equity and crawlers abandon long ones. And keep redirects live for years, because old links keep arriving forever.

The migration playbook: export the complete old URL inventory from a crawler plus Search Console plus analytics plus your most-linked pages. Build a one-to-one redirect map for every URL with traffic, links or rankings. Stage the new site and verify it crawls cleanly, checking that staging's robots block does not ship to production. Cut over, then immediately verify redirects, robots.txt, sitemaps, canonicals and analytics. Monitor for four to eight weeks and expect a dip measured in weeks. A dip measured in months means unmapped URLs.

One rule above all: never combine a domain change, a redesign and a platform move in one weekend. When traffic drops you'll have no way to isolate which one did it. That discipline is most of what makes a rebuild keep its rankings instead of restarting from zero.

What to actually do with this

Work it as an audit, in order: robots.txt, then sitemap, then indexation, then HTTPS, then mobile parity, then Core Web Vitals, then redirects. That order is deliberate. It follows the pipeline from crawl to index to rank, so you find the fatal problems before you spend time on the marginal ones.

All of it ships as standard on the sites I build, at no extra cost, because it's nearly free at build time. If you want to know where your current site stands, the free audit covers most of this list and emails you a scored report.

Common questions

What is a technical SEO audit?

A systematic check of everything that affects whether search engines can reach, understand and trust your site: robots.txt, XML sitemaps, indexation and canonicals, HTTPS correctness, mobile parity, Core Web Vitals, JavaScript rendering and redirects. Work through it in that order, because it follows the crawl-index-rank pipeline and surfaces the fatal problems before the marginal ones.

What are good Core Web Vitals scores in 2026?

LCP at 2.5 seconds or less, INP at 200ms or less, and CLS at 0.1 or less, all measured at the 75th percentile of real Chrome users. INP replaced FID in 2024. Chase the field data in Search Console rather than your Lighthouse lab score, because field data is what Google actually assesses.

Should I block AI crawlers like GPTBot in robots.txt?

For most businesses, no. Blocking GPTBot, ClaudeBot and PerplexityBot keeps your content out of AI training data, but it also removes you from the answer engines those systems power. You cannot be cited by an engine that cannot read you. Also allow the search-time fetchers, OAI-SearchBot and Claude-SearchBot, which power live browsing.

Does robots.txt stop a page appearing in Google?

No. Robots.txt blocks crawling, not indexing. A blocked URL can still appear in results, listed by URL alone, if other sites link to it. To keep a page out of the index, use a noindex meta tag and allow crawling so Google can actually see the tag.

Will a website redesign hurt my SEO?

It will if you skip the redirect map. Export every old URL with traffic, links or rankings, map each to its closest new equivalent with a single-hop 301, and verify after cutover. Expect a dip measured in weeks; a dip measured in months means URLs went unmapped. Never combine a domain change and a redesign.