SEO works in two connected layers. Search engines run every page they find through three stages automatically — crawling, indexing, and ranking. “Doing SEO” is what a site owner or practitioner does to influence how a page performs at each stage, through content, technical structure, and the credibility signals a site earns from the rest of the web.
That two-sided split is the whole mechanism. Every SEO tactic you’ve heard of — a , a fixed sitemap, an earned backlink — is an attempt to move one of those three stages in your favor. The sections below walk through each stage, then the practical process built on top of it.
How Do Search Engines Crawl a Site?
Crawling is the discovery step. Search engines use automated programs — bots or spiders, Googlebot being the best-known — that move page to page by following links, building an ongoing map of the web.
A few things determine how well a site gets crawled:
Internal and external links. Bots mostly find new pages by following links from pages they already know. A page with nothing linking to it — an “orphan” page — is much harder to discover, even if it exists.
XML sitemaps. A file listing a site’s pages directly, giving bots a more reliable map than link-following alone. It doesn’t guarantee crawling or indexing, but helps bots find pages they might otherwise miss.
Robots.txt. Tells bots which parts of a site they can access. A misconfigured robots.txt file can accidentally block bots from important pages — a common, costly mistake.
Crawl budget. Search engines don’t crawl every page with equal frequency. Larger sites especially have a practical limit on how much gets crawled in a given period, so low-value pages can eat attention that should go to the pages that matter.
Crawling only tells a search engine a page exists. It doesn’t mean the page gets indexed or ranked — those are separate steps.
How Do Search Engines Decide What to Index?
Indexing is where a crawled page either gets added to the search engine’s database — the index — or gets left out. Being crawled isn’t the same as being indexed; a search engine can know a URL exists and still decide not to store or serve it.
A few things happen here:
Rendering. The engine processes a page’s HTML, CSS, and JavaScript to see what a visitor would actually see. Content that only appears after heavy JavaScript execution can be missed if the page isn’t built to render cleanly.
Understanding the content. The engine analyzes what the page covers — topic, structure, the concepts it references — and stores that understanding alongside the page.
Canonicalization. When a search engine finds multiple URLs with the same or similar content — common with e-commerce filters, tracking parameters, or www versus non-www — it picks one version, the canonical, to index and treats the rest as duplicates.
The directives you control. A `noindex` tag tells a search engine “you can access this page, but don’t add it to the index” — different from blocking it in robots.txt, which tells bots not to request it at all. A `noindex`-ed page can still be crawled and evaluated; a page blocked in robots.txt may never be evaluated at all. Confusing the two is a common cause of SEO problems.
Pages also get left out for practical reasons: thin content, duplicates with no clear canonical, or content the engine judges isn’t useful enough to serve. Getting indexed doesn’t guarantee ranking well, either.
How Do Search Engines Rank Indexed Pages?
Ranking happens the moment someone actually searches. The engine looks across its index, pulls pages relevant to the query, and orders them using an algorithm that weighs many signals at once.
Search engines don’t publish the exact formula, and the weighting shifts over time and by query type. The signal categories are generally understood, though:
Relevance. How well a page matches what the searcher is looking for — not just matching words, but matching intent. See How to Use Keywords for SEO for how keyword research feeds this stage.
Authority and trust. Backlinks from credible sites, a domain’s reputation, and demonstrated topical expertise — built mostly off the page itself.
Technical health and page experience. Site speed, mobile usability, secure connections, and stability metrics like . A slow or broken page has a ceiling on how well it can rank.
Content quality and structure. Clear organization and thorough coverage, built to genuinely help the searcher rather than to attract the search engine.
Freshness, where relevant. For time-sensitive queries — news, “best of” lists that change — how recently a page was updated tends to carry more weight than for evergreen topics.
Worth flagging: some commonly repeated “ranking factors” aren’t confirmed. Dwell time — how long someone stays on a page before returning to results — is often cited as a signal, but it isn’t a confirmed part of Google’s algorithm. Lists claiming “200 ranking factors” mix real signals with speculation.
What Is the Basic SEO Process, Step by Step?
Everything above is what a search engine does on its own. The SEO process is what a practitioner — or a business owner doing it themselves — does to influence that deliberately. Stripped of agency branding, the basic sequence looks roughly like this:
1. Audit the current state. What’s indexed, what’s ranking, what technical issues exist, and where the content gaps are.
2. Research keywords and topics. Identify what your audience actually searches for, and the intent behind it, rather than guessing.
3. Fix technical issues. Crawlability, indexability, site speed, mobile usability — the infrastructure everything else depends on.
4. Create or improve content. Build pages that directly and thoroughly answer the target searches, organized with clear headings.
5. Build internal links. Connect related pages so visitors and search engines both understand how the content fits together.
6. Earn authority signals. Backlinks from credible sites, primarily, along with brand mentions and citations elsewhere on the web.
7. Monitor and adjust. Track what’s ranking and what’s changing, then repeat the cycle on what needs it most.
That sequence holds whether you’re doing it yourself — see How to Do SEO Yourself — or paying someone else to run it; see What Is an SEO Service? for what a paid engagement covers. Who’s doing the work changes the depth and the tools. It doesn’t change the steps.
How Do AI Answer Engines Fit Into How SEO Works?
One layer sits on top of the crawl-index-rank model above: AI answer engines like ChatGPT, Google’s AI Mode, and Perplexity don’t return a ranked list of links — they generate a direct answer, sometimes citing sources and sometimes not. That’s a meaningfully different mechanism, and it’s still evolving.
These systems generally still depend on content that’s been crawled and understood somewhere on the web. But what makes content likely to get pulled into a generated answer isn’t identical to what makes it rank well traditionally — clear, direct answers to specific questions, and content organized so a single passage stands on its own, both appear to help, though exactly how each system weighs this internally isn’t publicly documented. The crawl-index-rank mechanics above still matter — content a crawler can’t find can’t be cited either — but they’re no longer the only mechanism worth building for.
Common Questions
What is SEO, in one sentence?
SEO — — is shaping a website’s content, structure, and reputation so search engines can crawl, index, and rank it well for the searches you want to be found for. For the fuller definition, see What’s SEO?
How is SEO calculated?
It isn’t, in the sense of a single published formula or score. Search engines weigh many signals — relevance, authority, technical health, content quality — through an algorithm that isn’t public and changes over time. Some third-party tools generate their own composite “SEO score” as a diagnostic shortcut, but that’s the tool’s calculation, not a figure Google discloses.
How long does the SEO process take to actually show results?
Typically months rather than weeks, especially for a newer site with little existing authority. Crawling and indexing happen quickly once a page is live and linked; ranking — competing against everything else already indexed for that query — takes longer, particularly the authority signals earned backlinks provide.
Can a page be indexed but still not rank well?
Yes — a common source of confusion. Indexing means the engine has stored and understood the page. Ranking is a separate, ongoing competition against every other indexed page relevant to the same query. A page can be fully indexed and still sit far down the results if it’s thin, slow, or has little authority behind it.
What stops a page from being crawled in the first place?
The most common causes: no links pointing to the page (an orphan page), a robots.txt file blocking the bot, a broken site structure bots can’t navigate, or server errors when the page fails to load on request. Crawl issues like these are usually the first thing a audit checks.
Do search engines and AI answer engines run the same process?
Not exactly. Both generally depend on content having been crawled and indexed somewhere, but traditional search ranks a list of pages against a query, while AI answer engines generate a single response that may or may not cite sources. See the AI section above for how that plays out.