Technical SEO
& performance audit
in Austin

A site serving several countries or languages needs to tell a crawler which version belongs to which audience. That signal is narrow, frequently misconfigured, and regularly asked to do a job it cannot do.

What it does: it groups alternate versions of the same page so the right one is shown to the right audience. What it does not do: make a page rank where it otherwise would not. A thin translated page does not rise because it is correctly labelled.

The rules are unforgiving in a specific way. The annotation has to be reciprocal — if A points at B, B points back at A, or the pair is ignored. Every version has to list itself. The codes have to be valid, and a common error is inventing a region code that does not exist.

A default fallback is worth setting for language variants. Then a visitor outside every listed region lands somewhere deliberate. Not somewhere arbitrary.

Automatic redirection by location causes more damage than it prevents. A crawler arriving from one country and redirected to a local version never sees the others, and a traveller cannot reach their own. Suggest, do not force.

Near-duplicate variants are the other trap. Two pages in the same language for neighbouring markets, differing only in a currency symbol, are close enough that one will be chosen and the other ignored. If the content does not genuinely differ by market, one page usually serves both better.

Canonical tags and the language annotation have to agree. A page pointing its canonical at another language contradicts its own grouping. The contradiction resolves one way: the signal gets discarded.

What we check and report: reciprocity across every pair, code validity, the fallback, agreement with canonicals, and whether the intended version is the one actually appearing per market.

An Austin team that needs its crawl and index directives sorted out gets an audit that maps every rule and fixes the conflicts. Pricing opens with the ~$600 audit-and-fixes tier, and the full technical setup is ~$1,500. How long it runs depends on the site. We write the timeline into the estimate after a first look.

Three mechanisms control crawling and indexing. They do different jobs. Robots.txt decides whether a crawler may fetch a URL. The meta robots tag decides whether a fetched page may be indexed. The X-Robots-Tag header does the same for any file, including PDFs and images, where a meta tag cannot live.

Trouble starts when they overlap. A page blocked in robots.txt cannot be fetched, so Google never sees the noindex placed on it. The URL can still enter the index if other pages link to it, listed with no snippet. Teams add both rules to be safe and get the opposite result.

Order of removal matters. To drop a set of pages from the index, leave them crawlable with noindex until they fall out. Block them afterwards, if at all.

Robots.txt has its own traps. The file is read per host, so every subdomain needs one. Rules match by path prefix. One missing slash can block a whole directory. Google stopped honouring a noindex line inside robots.txt in 2019, and some old files still rely on it. Blocking CSS or script folders stops Google from rendering the page as a visitor sees it.

Headers hide the rest. A CDN or server rule can add a noindex header to an entire path. Nothing in the page source shows it. We read response headers for a sample of every template, since a crawl tool set to skip headers will miss this.

Internal nofollow is the last item. Google treats it as a hint. On navigation links it mostly cuts pages out of the link graph you built on purpose.

When a page carries both a generic robots tag and a googlebot tag, Google combines them. The most restrictive value wins. Snippet controls live in the same tag: nosnippet, max-snippet, max-image-preview. A theme that sets max-image-preview to none keeps large thumbnails out of Discover. Nobody notices, because the page still ranks.

Testing is cheap. The robots.txt report in Search Console shows which version Google fetched and when. URL Inspection shows whether a given URL is allowed and which directive applied. We test each new rule against a list of real URLs before it ships.

The deliverable is a directive map. Per template it lists the robots.txt status, meta robots value, header value and intended state. Each mismatch comes with the one-line change that fixes it.