serkankaya.
Türkçe sürüm
Let’s work together ↗

What Is Technical SEO? An End-to-End Guide from Crawl to Index

What is technical SEO and what does it cover? A guide to crawling, indexing, canonicals, JavaScript, Core Web Vitals and structured data, plus a checklist.

The short answer to what is technical SEO: technical SEO is the set of infrastructure work that makes sure a website can be crawled, rendered and indexed by search engines without problems and served quickly to users. It is not concerned with what the content says, but with how the content reaches the search engine. Without a solid technical foundation, even the best content stays invisible.

In this article I cover technical SEO end to end, from crawling and indexing to speed and structured data. Coming from a software development background, I look at problems from the code and server side. I describe how I apply this approach in more detail on my SEO services page.

Why is technical SEO important?

Technical SEO is important because if Google cannot crawl or index a page, that page cannot rank for any query. According to Google’s own guide, Google Search works in three stages: crawling, indexing and serving search results. The same guide clearly states that not every page makes it through all three stages.[1]

This structure also sets the priority in SEO. Content and links come into play at the ranking stage. Technical SEO makes sure the page can get that far. In short, technical SEO does not bring rankings by itself, but it removes the obstacles in front of them.

In the same guide, Google also says it does not accept payment to crawl a site more frequently or rank it higher.[1] So you cannot buy crawl frequency or quality. You can only influence it by building the site’s infrastructure properly.

Crawling Rendering Indexing Ranking
A page's path to the search results: technical SEO removes obstacles in the first three stages, while content and authority decide the ranking.

The difference between technical SEO, content SEO and off-page SEO

SEO is usually discussed under three headings. They are not alternatives to each other; they are different layers of the same goal.

LayerCore questionTypical work
Technical SEOCan the search engine reach and understand the page?Crawling, indexing, canonicals, speed, rendering, schema
Content (on-page) SEODoes the page answer the search intent well?Topic coverage, headings, internal links, title and meta
Off-page SEODo other sources find the site trustworthy?Links, digital PR, brand mentions

I covered the content side in how to write SEO-friendly content and the keyword side in my keyword research guide.

What does technical SEO cover?

Technical SEO looks for answers to four main questions: can the page be found, is it consolidated on a single address, is it understood correctly and does it load quickly? The sections below address each of these four questions with concrete settings.

Crawling is the process in which a search engine bot such as Googlebot discovers and downloads the URLs on a site. The bot finds new URLs in two main ways: links on the site and on external sites, and the XML sitemap file. That is why an important page needs to appear both in the sitemap and in internal links.

robots.txt is a text file at the root of the site that tells bots which paths they should not crawl. According to Google’s documentation, this file is used mainly to avoid overloading the site with requests and is not a mechanism for keeping a page out of Google.[2] A URL blocked by robots.txt can still appear in results without a description if other sites link to it.

A common mistake I see is blocking filter and search pages in robots.txt while also trying to give them a noindex. Because the bot cannot crawl the page, it never sees the noindex directive. I covered robots.txt rules for AI bots separately in AI crawlers and robots.txt.

The internal link structure is also part of crawling. Pages that sit too many clicks away from the homepage, or that receive no links from any page, are discovered late. I covered this in detail for online stores in ecommerce site architecture.

Indexing: noindex, status codes and canonicals

Indexing is the process in which the content of a crawled page is processed and added to Google’s database so it can be shown in search results. Not every crawled page is indexed. Google decides by looking at duplicate content, low quality and canonical signals.

If you do not want a page to be indexed, you use noindex. Noindex can be given with a meta robots tag or an X-Robots-Tag HTTP header. Google’s robots.txt guide also recommends noindex or password protection to keep a page out of the results.[2]

HTTP status codes also affect indexing directly. A healthy page returns 200. A page that has moved permanently is redirected to its new address with a 301. A page that has really been removed returns 404 or 410. Pages that return 200 even though they are empty are flagged as “soft 404” in Search Console.

The canonical tag is used to specify the main version among URLs that are very similar to each other. For example, addresses with ?utm_source= parameters, www and non-www versions, or colour-filtered product pages can show the same content. According to Google’s documentation, redirects and rel="canonical" are strong signals, while sitemap inclusion is a weak signal.[3]

<link rel="canonical" href="https://example.com/product/leather-bag/" />

The point to remember here is this: a canonical is not a command but a suggestion. If other signals conflict, Google may choose a different URL as canonical.[3] That is why the canonical, internal links, the sitemap and redirects should all point to the same URL.

Rendering: how does Google see JavaScript pages?

Short answer

Can Google see content loaded with JavaScript?

Yes, but with a delay and under conditions. Google processes pages that use JavaScript in crawling, rendering and indexing phases. If content only appears after JavaScript runs in the browser, or if a resource is blocked or fails during rendering, Google may not see part of the content. Generating important content on the server reduces this risk.

Rendering is the process in which a browser or search engine runs HTML, CSS and JavaScript to build the final version of the page. According to Google’s JavaScript SEO guide, Google processes JavaScript web apps in three main phases: crawling, rendering and indexing.[4] The rendering phase can take place in a separate queue after crawling.

This difference matters in single-page applications (SPAs) built with React, Vue or a similar framework. If the initial HTML response only contains an empty container, the headings, text and links appear only after rendering. Server-side rendering (SSR) or static site generation (SSG) removes this dependency by putting the content into the initial HTML response.

The check is simple. You run the live test in Search Console’s URL Inspection tool and look at the rendered HTML. If the text and links you see on the page are not there, Google may not be seeing them either. Links also need to be given as <a href="…">; links that only work through onclick cannot be crawled.

Speed and user experience: Core Web Vitals

Core Web Vitals are three metrics that Google uses to measure page experience in terms of loading speed, responsiveness to interaction and visual stability. These metrics are called LCP (Largest Contentful Paint), INP (Interaction to Next Paint) and CLS (Cumulative Layout Shift). Google states that keeping these metrics good helps both search performance and user experience.[5]

LCP: loading of the largest content
≤ 2.5 s
INP: response time to interaction
< 200 ms
CLS: visual shift score
< 0.1
Source: Google Search Central: Core Web Vitals

These thresholds are the “good” values in Google’s Search Central documentation.[5] The Core Web Vitals report in Search Console classifies URL groups against these thresholds based on real user data.

In practice, speed problems are usually solved in the template, not on a single page. For LCP, you need to reduce the size of the largest image, serve it in the right format (WebP or AVIF) and avoid lazy loading it. For INP, reducing heavy third-party tags and breaking up long JavaScript tasks is effective. For CLS, giving images and ad slots a width and height in advance is enough.

Working on the performance marketing side, I have also seen this: every new pixel added to the tag manager makes INP a little worse. Moving part of the measurement from the browser to the server helps both data quality and speed. I explained this approach in what is server-side tracking.

Understanding: structured data and site structure

Structured data is markup that tells a search engine the type of information on a page (product, article, person, organization, FAQ) in a machine-readable form. Its most common form is JSON-LD written with the Schema.org vocabulary. Structured data does not raise rankings directly, but it makes a page eligible for rich result displays and clarifies what the page is about.

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Article",
  "headline": "What Is Technical SEO?",
  "author": { "@type": "Person", "name": "Serkan Kaya" },
  "datePublished": "2026-11-19"
}
</script>

The other half of being understood is site structure. A logical URL hierarchy, breadcrumb navigation and consistent internal links show the relationships between pages to both users and search engines. On multilingual sites, hreflang tags indicate which page is meant for which language and region.

Technical SEO is also the foundation of AI-powered search systems, because these systems also have to crawl and read a page first. If ChatGPT, Perplexity or Google’s AI Overviews are going to use a page as a source, that page needs to be accessible and readable.

Three technical details stand out here. First, AI bots should not be blocked by mistake in robots.txt. Second, content should be present in the initial HTML response without depending on JavaScript; some bots read a page without running JavaScript. Third, changes should be announced quickly. The IndexNow protocol can be used to notify search engines such as Bing and Yandex of changes.

I covered the strategic side of the topic in what is GEO, and the idea of offering a site summary to AI systems in what is llms.txt.

How do you do a technical SEO audit?

Short answer

How do you do a technical SEO audit?

A technical SEO audit is done by comparing Search Console data with the output of a crawling tool. First you look at how Google sees the site, then at how the site presents itself. The differences between the two views show the issues that need fixing. The issues are then ranked by the number of pages affected and business impact.

  1. Review the Search Console page indexing and Core Web Vitals reports
  2. Crawl the site with a crawler such as Screaming Frog
  3. List status codes, redirect chains and broken links
  4. Compare canonical, noindex and sitemap consistency
  5. Check the rendered HTML of important templates
  6. Prioritize issues by impact and effort

Tools help in an audit, but they do not make the decisions. Tools such as Screaming Frog, Semrush, Ahrefs or Sitebulb can produce thousands of warnings. Most of these warnings are not real problems. What matters is telling apart the issues that affect the pages that should rank.

When prioritizing, I follow this order: first pages that should be indexed but are not, then pages where the wrong URL was chosen as canonical, and then template-level speed and rendering issues. Small warnings such as individual meta description lengths go to the end of the list.

If you are looking for a checklist specific to online stores, see the ecommerce SEO checklist. The list below contains the core items you can use for a pre-release or monthly check on any type of site.

Technical SEO core checklist0/10

When is technical SEO a priority?

Every site needs technical SEO, but in some situations it moves to the top of the priority list. What these situations have in common is a large number of pages or a large infrastructure change.

  • Site migrations and redesigns: If no redirect map is prepared when the domain, URL structure or CMS changes, traffic can be lost.
  • Large ecommerce sites: Filter, sort and variant URLs can produce thousands of duplicate pages. On these sites, crawl budget and canonical management become critical.
  • JavaScript-heavy sites: Rendering issues are common in applications that build content in the browser.
  • Multilingual and multi-market sites: If hreflang, language folders and regional redirects are set up incorrectly, the wrong page shows up in the wrong country.
  • Indexing problems: If the number of non-indexed pages in Search Console is rising quickly, you need to find the cause before producing more content.

In these situations, technical SEO is usually carried out together with a developer. Being able to describe the problem at code level determines how fast it gets fixed. I described the work I do on the web development side on my software development page.

Conclusion: where should you start with technical SEO?

Technical SEO is the infrastructure that lets content reach the search engine: crawling, rendering, indexing and speed. The best place to start is Search Console. If there are important non-indexed URLs in the page indexing report, poor groups in the Core Web Vitals report and wrong canonical choices in URL Inspection, your first to-do list is ready.

The next step is to crawl the site with a crawling tool, compare it with the Search Console view and rank the issues by business impact. Running the checklist above before every major release and design change catches problems before they grow. If you are curious about how I work, take a look at my about page.

Key terms

Technical SEO
Technical SEO is the set of infrastructure work that makes sure a website can be crawled, rendered, indexed and served quickly by search engines. robots.txt, sitemaps, canonicals, redirects, rendering, site speed and structured data are its main subjects.
Canonical tag
Canonical tag is the rel="canonical" link element that tells a search engine which of several URLs with identical or very similar content is the main version. Google treats it as a strong signal, but it may choose a different URL as canonical based on other signals.
Rendering
Rendering is the process in which a browser or search engine runs a page's HTML, CSS and JavaScript to build the final version the user sees. On sites that load content later with JavaScript, a search engine may miss part of the content until rendering is complete.
Noindex
Noindex is a directive, given with a meta robots tag or an X-Robots-Tag HTTP header, that asks for a page not to be shown in search results. For Google to see this directive, the page must not be blocked from crawling in robots.txt.
Redirect chain
Redirect chain is a situation where a URL reaches its destination not in a single step but through several intermediate redirects. Chains waste crawl resources, slow down page loading and commonly appear after site migrations.

Frequently asked questions

What is technical SEO?

Technical SEO is the set of infrastructure work that makes sure a website can be crawled, rendered and indexed by search engines without problems and served quickly to users. It is not concerned with the content itself, but with the path that content takes to reach the search engine.

What is the difference between technical SEO and content SEO?

Content SEO deals with what a page says and how well it answers the search intent. Technical SEO deals with whether that content can be found, consolidated on a single canonical address and loaded quickly. No matter how good the content is, a page that cannot be crawled cannot rank.

How often should technical SEO work be done?

A full technical SEO audit is usually done a few times a year, but continuous monitoring matters more. Checking Search Console reports weekly and running a short checklist before every major release, redesign or site migration catches most problems before they grow.

Does robots.txt hide a page from Google?

No. According to Google's documentation, robots.txt is used mainly to avoid overloading a site with requests and is not a mechanism for keeping a page out of Google. To keep a page out of the index, you need to use noindex or password-protect the page.

Can I do technical SEO without knowing how to code?

Reading Search Console reports, checking robots.txt and the sitemap, and scanning for broken links can all be done without coding. Rendering issues, template-level canonical errors and performance improvements, however, are usually solved together with a developer. A good technical SEO specialist should be able to describe the problem at code level.

Sources

  1. Google Search Central: In-depth guide to how Google Search works
  2. Google Search Central: Introduction to robots.txt
  3. Google Search Central: How to specify a canonical URL
  4. Google Search Central: Understand JavaScript SEO basics
  5. Google Search Central: Understanding Core Web Vitals and Google search results