HTTP status codes for SEO: 404, 410, soft 404, 5xx and 429

When a crawler requests a URL, the server answers with a three-digit status code before it sends a single byte of the page. That number tells the crawler how to treat everything that follows: index this, follow this redirect, forget this page, come back later, or slow down. Search engines take the code at face value most of the time, and when the code and the content disagree, they have to guess, which is where many indexing problems come from.

This guide covers the codes that matter for SEO, what search engines do with each, and the misconfigurations that show up again and again in audits.

The codes at a glance

Code	Meaning	What search engines do
200 OK	Here is the page	Process the content; it may be indexed
301 / 308	Moved permanently	Follow; transfer signals to the target; eventually index the target instead
302 / 307	Moved temporarily	Follow; keep the original URL indexed at first; treated as permanent if it lasts
304 Not Modified	Unchanged since your last visit	Reuse the stored copy; saves bandwidth for both sides
404 Not Found	No page here	Drop from the index after a few confirmations; keep checking occasionally
410 Gone	Removed on purpose	Drop from the index, possibly a little faster than 404
401 / 403	Not allowed	Not indexed; for Google, treated much like a 404 for indexing
429 Too Many Requests	Slow down	Reduce crawl rate; Google treats it like a server error
500, 502, 503, 504	Server problem	Retry later; reduce crawl rate; drop pages if errors persist

Redirects have their own guide: redirects, HTTPS and www covers 301 versus 302, chains and loops. This article focuses on the other codes.

404 and 410: removing pages properly

A 404 is the correct response for a URL that has no content. Sites remove pages all the time: discontinued products, expired events, outdated articles. Returning 404 for them is healthy, and Google has repeatedly said that 404s on their own do not harm a site's rankings.

The decision that matters is whether to return 404 or redirect. A simple rule works well:

• Redirect if there is a close replacement a visitor would be happy to land on: the new version of a product, the merged article, the parent category of a discontinued item.
• Return 404 or 410 if there is no relevant replacement. Redirecting everything to the home page is not a replacement; Google typically treats such redirects as soft 404s anyway.
• Keep the page with a clear status message if it still has value to visitors, as with seasonal products that will return. The out-of-stock products guide covers that case in detail.

Whichever you choose, remove internal links to removed pages and take them out of your XML sitemaps. A 404 that your own navigation still links to is a broken link for users and wasted crawling for bots.

A good 404 page helps users without confusing crawlers: it returns the 404 status, explains that the page does not exist, and offers search and links to main sections. It does not redirect to the home page and does not return 200.

Soft 404s: when 200 lies

A soft 404 is a page that returns 200 but looks to a search engine like an error or an empty page. Google detects them from the content and reports them in Search Console's Page indexing report under "Soft 404". Common causes:

• Custom "page not found" templates that return 200 instead of 404.
• Empty category or search result pages: "0 products match your selection".
• Product pages for discontinued items stripped down to a name and "no longer available".
• Pages whose main content loads with JavaScript and fails, leaving only a header and footer.
• Many unrelated URLs redirected to one generic page.

The fix depends on the cause. A not-found template must return 404. Empty listings should either be prevented (do not link to filter combinations with no results) or return 404 when empty. Thin discontinued product pages need either real content or a proper removal. And if a page is genuinely valuable but Google calls it a soft 404, look at it as a crawler would: view the raw HTML, check it renders, and make sure the main content is substantial and visible.

5xx errors: the expensive ones

Server errors are treated as a sign that your site is struggling. Googlebot responds by crawling less, to avoid making things worse. If errors persist, URLs that keep failing are eventually dropped from the index. Short outages of minutes rarely leave a trace; outages of hours, or intermittent errors on a share of requests over days, can reduce crawling for a while after the server recovers.

The most common real-world sources of 5xx responses for crawlers:

• Timeouts on slow pages. Heavy pages that take seconds to generate hit the proxy or PHP timeout under crawl load and return 502 or 504. See server response time.
• Rate limiting or bot protection that returns 503 or 429 to search engine IPs. Firewalls configured against scrapers sometimes catch Googlebot or Bingbot, especially after a rule change.
• Deploys that briefly break the application for every request.
• Database connection limits reached during crawl spikes.

Check the Crawl Stats report in Search Console, which breaks down responses by status, and your server logs, which show exactly which URLs failed and when.

503 for maintenance, done right

For planned downtime, return 503 Service Unavailable with a Retry-After header giving the number of seconds until you expect to be back. Crawlers will retry rather than index your maintenance page. Two cautions:

• Serve the 503 for every URL, including robots.txt. Be aware that Google treats a robots.txt that returns 5xx for a long time as a reason to stop crawling the site. A few hours is fine; days is not.
• Do not leave a 503 in place for more than a day or two. Prolonged 503s eventually lead to pages being dropped, just like other server errors.

429 and bot protection

A 429 Too Many Requests tells a client to slow down. Google treats 429 like a server error and lowers its crawl rate. That makes it a blunt tool: it works for throttling a crawler that is genuinely overloading your server, but if a firewall or CDN starts sending 429s to Googlebot because of a misconfigured rule, crawling of the whole site can drop. After any change to bot protection, check your logs for verified search engine bots receiving 403, 429 or 503 responses.

Status codes that are easy to forget

Pages are not the only URLs with status codes that matter.

• robots.txt. A 404 means "no rules, crawl everything", which is fine. A 5xx means "something is wrong", and Google may pause crawling of the whole site until it can read the file. If robots.txt sits behind the same bot protection as the rest of the site, a firewall rule can stop all crawling at once. The robots.txt mistakes guide covers this in more depth.
• XML sitemaps. A sitemap URL that returns 404 or an HTML error page is reported in Search Console as "Couldn't fetch". Sitemaps generated on the fly by plugins sometimes time out on large sites and return 5xx under load.
• CSS, JavaScript and API endpoints used for rendering. If they return errors to crawlers, the rendered page can look empty, which leads straight back to soft 404s.
• Images. Product images that return 403 to unknown user agents, often due to hotlink protection, never appear in image search.
• HEAD versus GET. Some servers answer HEAD requests differently from GET. Crawlers use GET, so always test with GET when diagnosing.

How to audit status codes

1. Crawl the site and list every internal link whose target returns anything other than 200. Fix links to 3xx and 4xx targets at the source.
2. Check the Page indexing report for "Not found (404)", "Soft 404", "Server error (5xx)" and "Blocked due to access forbidden (403)". Inspect a sample of each. The Page indexing report guide explains each status.
3. Request a made-up URL on your domain, such as /this-page-does-not-exist-123, and confirm it returns 404, not 200 or a redirect.
4. Look at logs for 5xx and 429 responses to verified crawlers, grouped by hour, to spot patterns around deploys or traffic peaks.
5. Monitor continuously. Status problems tend to appear suddenly after a change. The SEO checker reports the status of your home page and redirects, and scheduled checks alert you when a page that returned 200 starts failing.