Content & evidence / Lesson 04
Technical GEO: Crawlers, Canonicals, Structured Data, and llms.txt
Separate the technical controls that affect discovery from the optional files and myths that often distract GEO teams.
Tim WalshAdvanced13 min readUpdated 2026-07-13Learning objective
Choose the correct technical control for crawling, indexing, canonicalization, entity markup, and AI-search access.
A technical GEO audit ordered by impact, with unsupported shortcuts kept out of the critical path.
Use each technical control for the job it actually performs
Technical control map
Confusing these controls can hide content or split signals instead of improving visibility.
| Control | Primary job | Common mistake |
|---|---|---|
| robots.txt | Manage crawler access to paths | Using it as a reliable noindex or canonicalization method |
| robots meta / X-Robots-Tag | Control indexing and snippets for accessible resources | Blocking the crawler before it can read the directive |
| rel=canonical | Suggest the representative URL for duplicate content | Pointing similar pages at conflicting canonicals |
| sitemap | Declare important canonical URLs and update signals | Listing redirects, errors, or non-canonical variants |
| structured data | Describe visible entities and content in a standard format | Marking up information that users cannot see |
Give every important claim a stable home
Canonicalization consolidates duplicate or near-duplicate URLs around a representative page. For GEO work, that stable page becomes the place where the full claim, author, update date, evidence, and internal links can accumulate.
Structured data can provide explicit clues about the entities and content on that page. It should mirror visible information and use complete, accurate properties rather than attempting to label claims that the reader cannot verify in the page body.
A stable evidence page
Keep technical signals and visible content aligned.
- 01
Canonical URL
Choose one durable page for the topic, product, policy, or report.
- 02
Visible evidence
Publish the claim, explanation, dates, sources, tables, and limitations in HTML.
- 03
Structured identity
Add JSON-LD that accurately describes the visible article, organization, author, or product.
- 04
Connected context
Link related guides, product pages, reports, and methodology from the canonical page.
Keep llms.txt outside the critical path
Google explicitly states that llms.txt and other special AI text files are not required for its generative AI search features and do not improve Google Search visibility. Other services may document their own crawler controls or machine-readable conventions, so the answer is platform-specific rather than universal.
A team may still maintain an llms.txt file for systems that use it, but the file should point toward strong canonical pages. It should not replace accessible HTML, source-backed claims, clear navigation, or vendor-supported crawler configuration.
Technical GEO priority order
Fix the conditions that affect real discovery before optional support files.
- Resolve status-code, rendering, robots, noindex, authentication, and WAF barriers.
- Consolidate duplicate URLs and keep internal links aligned with the canonical choice.
- Publish complete visible evidence with descriptive headings and useful text alternatives.
- Validate structured data against the visible page and the relevant platform guidance.
- Add optional machine-readable support only when a target system documents how it is used.
Geolity keeps technical findings tied to buyer impact
Geolity in practice
Move from a crawl finding to an optimization priority
Geolity's GEO Optimization workflow evaluates page access, content and entity clarity, source evidence, and implementation readiness. The report keeps those findings beside prompt evidence so teams can prioritize technical work that affects real buyer questions.
- Identify challenge pages, weak public evidence, and page-readiness gaps.
- Connect a technical issue to the prompts and answers where the brand is weak or absent.
- Turn findings into high, medium, and low priority actions with implementation guidance.
- Use later reports to compare whether the same page and prompt family improved.
Questions from this lesson
Is structured data required for generative AI citations?
No universal requirement exists. Structured data can clarify visible entities and enable supported search features, but it does not guarantee retrieval, ranking, or citation.
Should every site publish llms.txt?
Only when a target system or workflow gives the file a defined purpose. It should remain secondary to accessible canonical pages and vendor-supported crawler controls.