A Stockholm software company usually has no idea how many URLs it publishes. The marketing site has forty. The documentation has four thousand, most of them generated. Nobody decided this, and nobody is measuring which of them a crawler has ever seen.
Indexing is the least glamorous part of search work and the part that most often decides the outcome. A page that is not in the index cannot rank at any position, no matter how well it is written. Yet indexing is treated as a background process that presumably takes care of itself — right up to the week someone notices that the new pricing page has been live for two months and returns nothing on a site: query.
Your estate publishes more pages than your team writes
Count what a mid-sized product company actually serves. A marketing site of thirty to sixty hand-written pages. A documentation site where every API method, every configuration key and every release note is a URL, frequently duplicated across two or three versioned trees. A changelog. A careers section where each open role has its own address, often on a subdomain belonging to the applicant-tracking vendor. A status page with a permanent URL per historical incident. An investor or press section.
The hand-written pages are the ones that get discussed, and they are a rounding error in the total. The generated ones dominate, they change without a person touching them, and they are produced by a build pipeline that has no opinion about search at all. Versioned documentation is the sharpest example: publish three supported versions and you have tripled the address space with near-identical text, and unless the canonical tags are right you have asked a crawler to work out which of three copies matters.
Discovery, crawling, indexing, serving
Most confusion in this area comes from one word doing four jobs. A URL passes through distinct states, and it can stop at any of them, permanently, without producing an error anywhere you would normally look.
| State | What has happened | What it does not mean | Where you see it |
|---|---|---|---|
| Discovered | The address is known to exist | That anything has been fetched | Sitemap and submission logs |
| Crawled | A bot requested the URL and got a response | That the response was usable | Per-URL visit log, server logs |
| Indexed | The content was stored and is eligible | That it will appear for any query | Search Console coverage |
| Serving | The page appears in results for something | That the position is worth having | Impressions in the analytics views |
The gaps between these states are where the work is. Discovered-but-never-crawled is usually a linking problem: the URL exists in a sitemap and nowhere else, so nothing signals that it matters. Crawled-but-not-indexed is usually a content or duplication problem — the page was fetched, evaluated and judged not worth storing separately, which is a decision, not a failure.
Crawl budget is a consequence, not an allowance
Crawl budget is often described as if a quota were handed out per site each morning. It is closer to a load-balancing outcome: how much a crawler is willing to fetch from your infrastructure, shaped by how fast you respond and how much of what it fetched last time turned out to be worth having.
That framing makes the practical question obvious. You do not get more budget by asking. You get more useful crawling by wasting less of what is already being spent, and there are a small number of reliable ways an engineering estate wastes it.
- Preview and branch deployments left public. Every pull request gets a hostname, every hostname is crawlable, and a year of merged branches sits there duplicating the production site.
- Parameter and fragment explosions. Documentation search pages, sortable tables and anchor-driven navigation can generate thousands of distinct addresses from one document.
- Redirect chains from three migrations ago. Each hop is a fetch. A four-hop chain spends four requests to deliver one page, and internal links usually still point at the first hop.
- Soft errors that return 200. A JavaScript route that renders an empty shell for a missing page is a successful response as far as the crawler is concerned, and gets fetched again and again.
None of these are exotic failures. They are the normal by-products of a fast-moving deployment culture, and they are invisible from the marketing side of the company because they live in infrastructure nobody thought of as a publishing surface.
Why the Swedish pages are the ones that never get crawled
On a site where English is the working language, the English pages have everything a crawler uses to decide something is important. They are linked from the documentation, from the careers pages, from the footer of every subdomain, and from whatever external coverage the company has picked up. The Swedish pages, added later, typically have one entry point from a language switcher.
The result is a specific, repeatable pattern: the Swedish translations sit in the sitemap, appear in the discovered state, and are crawled rarely or never. The team concludes that Swedish content does not work for them, and stops writing it. The actual cause was that nothing on the site pointed at those pages other than a switcher widget that many crawlers weight lightly.
Swedish page never crawled
Sits in the sitemap, no visit recorded in the log after weeks.
- Check inbound internal links
- Not a translation-quality issue
Crawled, then dropped
Fetched once, never stored, no coverage entry afterwards.
- Often a thin or duplicated page
- Check the canonical target
Docs outrank the product page
An API reference page takes commercial queries the marketing site wanted.
- Usually a linking imbalance
- Rarely worth fighting directly
Old domain still indexed
Pre-rename hostname serving live copies rather than redirecting.
- Check the redirect actually fires
- Watch for chains through www
The fix in every one of these cases is structural. Link the Swedish pages from Swedish pages, put them in the navigation rather than only in a switcher, and give them a reason to exist that is not a word-for-word copy of the English. If they are a copy, being crawled and dropped is the correct outcome, and no amount of submission will change it.
The Indexing Hub, and its actual limits
The Indexing Hub is the part of the panel that deals with this layer. It does three things: it submits URLs, it parses sitemaps, and it records what happened afterwards. The third of those is the one worth paying for, because it is the part that turns indexing from a belief into a measurement.
URL tracker and bulk submission
For estates where the URL count is measured in thousands rather than dozens.
- A daily budget of 1,000 URLs per account. This is the platform's own ceiling, and it forces the question of which thousand actually matter.
- Bulk submission up to 10,000 URLs per batch. Batches queue against the daily budget rather than bypassing it.
- Delivery through the IndexNow API. Submissions are passed to GoogleBot and BingBot through the protocol both support.
- Live counters for the batch. Submitted, found and failed are tracked separately, so a batch that silently half-worked is visible.
Sitemap submission and recursive parsing
For generated estates where the sitemap is itself an index of sitemaps.
- Upload a file or point at a URL. Either route feeds the same parser, which matters when the sitemap is generated at build time.
- Recursive parsing three levels deep. An index of indexes of sitemaps is resolved rather than treated as a single document.
- Up to 1,000 sitemaps in one job. Enough for a documentation tree that splits by version and by section.
- Two concurrent jobs, twenty queued. The queue is the reason to schedule large parses deliberately rather than firing them all at once.
Both of these are plumbing. Useful plumbing, and considerably faster than pasting addresses into a form one at a time, but plumbing. The part that changes how a team works is the log.
A log entry is worth more than a submission receipt
For each URL the Hub keeps a record of what happened to it: which bot visited, with a timestamp, what status came back, and the error detail when there was one. Alongside that, live counters track how many URLs in a batch were submitted, found and failed.
This turns a category of argument into a lookup. When someone says a page is not indexed, there is now a factual first question with a factual answer: has a bot ever fetched it, and what did it get? The three possible answers point in completely different directions, and none of them require guessing.
- No visit recorded at all. A discovery and linking problem. Submission may help; internal links will help more.
- Visited, non-200 status. An infrastructure problem with a specific error to hand to whoever owns the deployment.
- Visited, clean 200, still not indexed. A content or duplication judgement. Nothing technical left to fix; the page needs a reason to exist.
- Visited repeatedly, always the same empty shell. A rendering problem — the bot is being served a page your browser never shows you.
That third case is the one worth sitting with, because it is where most indexing effort is wasted. A page fetched cleanly and left out of the index has already been assessed. Resubmitting it changes nothing. If you have three versioned copies of the same API reference and only one is indexed, the system is behaving correctly and the fix is canonicalisation, not persistence.
A working order of operations
Given a finite daily budget and an estate of several thousand addresses, the sequence matters more than the volume. The order below is deliberately boring, and it front-loads everything that does not require submitting anything.
| Step | What you do | Why it comes here |
|---|---|---|
| 1 | List every hostname and who verified it | You cannot measure properties you have not connected |
| 2 | Close preview environments to crawlers | Removes waste before you spend anything |
| 3 | Parse the sitemaps and compare against the real URL count | The difference is usually the whole story |
| 4 | Fix redirect chains and soft errors found in the log | Cheap, and it changes crawl behaviour |
| 5 | Submit the commercially relevant pages only | The daily budget forces a choice; make it consciously |
| 6 | Read the visit log a week later | This is the measurement the rest of the work was for |
Step five is where most teams go wrong by submitting everything they have. A daily ceiling of 1,000 URLs against a documentation tree of 4,000 means four days of submissions, most of them spent on pages that will never receive a commercial query. Rank the list first. Our notes on the measurement tooling cover the reporting side of the same estate.
Pages with a commercial job
Pricing, product, comparison and the landing pages a campaign points at.
- Small list, high value
- Submit on the day they ship
Anything that changed materially
Rewritten pages, new release notes, a changed URL after a restructure.
- Change, not republish date
- Include the old URL's target
Swedish counterparts worth having
Only the ones with their own substance, not mirrored English text.
- Link them internally first
- Then submit
Generated reference pages
Per-method API pages, incident archives, expired job listings.
- Let crawling find them
- Spend the budget elsewhere
When to hand the loop over
Indexing work has a natural shape: an intense audit, then a low-grade recurring task. The audit is a person's job. The recurring part — watching for new URLs, submitting them, noticing when a batch fails — is exactly the sort of thing that gets skipped in a busy quarter and is better automated.
The campaign tiers sit alongside the Hub rather than replacing it. AutoSEO at 149 USD per month per domain runs keyword discovery, automatic backlink building from a partner network of over 230,000 websites, and on-site suggestions, with the full analytics views included. FullSEO at 500 USD per month per domain adds manual keyword selection with automatic fallback, directed backlink placement against a domain-rating target, a human review mode for on-site changes, and a team of specialists, developers and writers behind the automation. Add-on placements are priced per slot: Wikipedia at 10 USD in blocks of 0, 1, 5 or 10, PBN at 1 USD in blocks of 0, 20, 100 or 500.
Stream, the assistant in the My SEO area, is the practical bridge between the two. It accepts URL lists in batches, so a set of newly published documentation pages can go in as a list rather than a form-filling exercise, and it keeps newly placed backlinks, generated reports and to-dos in one chronological feed. A router model decides per question which data blocks to load — between zero and three of them by relevance — so an indexing question does not drag campaign data into the answer. You can read more about how the unified panel is put together, or look at the two campaign tiers side by side.
Frequently asked questions
We submitted the page a week ago and it still is not indexed. What now?
Check the visit log before doing anything else. If no bot has fetched it, the problem is discovery and internal linking. If a bot fetched it and got a clean response, the page has been assessed and left out, which is a content judgement — resubmitting will not change it.
Should we submit our entire documentation tree?
Usually not. With a ceiling of 1,000 URLs per day, submitting 4,000 generated pages spends four days on addresses that will never take a commercial query. Submit the pages you would be upset to lose, and let normal crawling find the rest.
Our preview deployments are indexed. How much does that hurt?
Mainly through waste and duplication: crawler capacity spent on copies, and several near-identical candidates for the same content. Block them at the host level rather than with a meta tag, since a tag has to be fetched before it can be obeyed. Then let the copies fall out naturally.
Does IndexNow work for Google, or only Bing?
Submissions through the Hub are directed at both GoogleBot and BingBot. What either engine does with the notification is its own decision. Treat IndexNow as a faster route to discovery, never as an indexing guarantee.
How deep can the sitemap parser follow an index of indexes?
Three levels, with up to 1,000 sitemaps in a single job. Two jobs run at once and up to twenty wait in the queue, so large documentation estates are worth scheduling rather than submitting all at once.
Indexing rewards inventory and evidence more than it rewards effort. Knowing which hostnames you publish, which URLs a bot has actually fetched, and what it received when it did, answers most of the questions that otherwise turn into a fortnight of speculation. The daily and per-batch ceilings are not obstacles so much as a forcing function: they make you decide which pages you actually care about.
If you want to see the current state of your own estate, the first useful step is connecting the properties and running a sitemap parse against them. You can start with a first domain in the panel and read the visit log before deciding whether anything needs submitting at all. Further pieces on technical work are collected in the English blog archive, and the platform's own the product site go into more depth on the Hub itself.
