A Vercel site that works for a signed-in team member can still present authentication to an anonymous visitor or crawler. Test the production URL without account cookies before changing robots.txt.
Check which URL is protected in Vercel Project Settings. Standard Protection excludes production domains, while All Deployments includes them. If public content shows a Vercel login barrier, review the protection scope for that project. Keep previews and private application routes protected, and verify the final production page anonymously.
This guide separates Vercel deployment access, application authentication, firewall rules and crawl directives. It explains how to diagnose an unintended public access barrier while preserving private previews and account areas.
Access evidence and the responsible setting
| Observed response | Investigate | Do not assume |
|---|---|---|
| Vercel authentication screen | Deployment Protection and URL scope | A robots.txt edit can remove authentication |
| Application sign-in screen | Route middleware and account rules | All application routes should become public |
| Firewall denial or challenge | Request event and matching security rule | A declared bot name proves identity |
| Public content with noindex | Response and HTML directives | A valid domain means indexing is allowed |
Main Explanation
Start with the exact URL reported by the crawler or audit. A generated deployment address, a branch preview and the custom production hostname are not interchangeable evidence. Write down the requested host, final destination, response status and visible screen. A report against an intentionally private preview should not trigger a change to public production security.
Repeat the request without your Vercel session or application cookies. Use a private browser window and, where practical, inspect the raw HTTP response. Team access can make a protected deployment look public during ordinary development. Keep a screenshot or redacted response excerpt so the person reviewing the configuration can identify whether the barrier belongs to Vercel or to your application.
Inspect Deployment Protection for the specific project and compare its scope with the current Vercel documentation. The setting describes which deployment URLs require access; it is not a robots directive. If a marketing site must be public, choose a configuration that allows its intended production domain while retaining the protection needed for development and review URLs.
If the barrier is an application sign-in page, inspect middleware and route guards instead. A broad authentication matcher may accidentally include /pricing, /docs or /blog. Explicitly identify which routes are public and test both sides of the boundary. Do not make account pages public or remove all authentication just because one landing page should be crawlable.
If the response is a firewall challenge or denial, inspect the corresponding request event and rule. A browser test and a crawler request can differ in source, headers and behavior. Use the provider identity evidence available in logs when deciding on exceptions. A familiar User-Agent alone is not sufficient proof that a request belongs to a trusted crawler.
Do not put a protection-bypass secret into a sitemap, canonical URL, public link or robots.txt. Public visitors and search systems should reach the ordinary canonical URL without a private credential. Use authorized bypass mechanisms only for the private automation they were designed for. Publishing a secret-bearing route can undermine protection and creates the wrong URL for discovery.
After correcting access, inspect robots.txt, the response X-Robots-Tag and HTML robots metadata independently. A public page can still carry noindex, while a private preview may appropriately retain exclusion controls. Also compare the canonical with the production host. These controls solve different problems; removing a login screen does not automatically correct a preview canonical or index directive.
Verify one public landing page, one public deep link and one private route after the change. The first two should deliver their intended content anonymously; the private route should still enforce access. Save that result with the deployment identifier. Then run a TruboRankAI scan against the production URL and use Search Console for Google-specific inspection. Treat later crawling or indexing as a separate observation rather than an immediate consequence you can guarantee.
Check the public Vercel deployment
Use the production URL to inspect observable discovery signals after confirming that private routes still require authentication.
Practical Steps
- Identify the exact production or preview URL.
- Capture an anonymous response.
- Locate the blocking layer.
- Adjust only the unintended public access restriction.
- Check directives and canonical URLs.
- Retest public and private routes, then inspect crawler evidence separately.
FAQ
Should my preview deployment be indexable?
Usually the canonical public production site is the discovery target. Keep previews limited to their review purpose, especially when they contain unfinished content or private data.
Can robots.txt allow a crawler through a login screen?
No. Robots rules express crawl preferences. They do not grant application or deployment access.
Why does a domain checker pass while a scan fails?
DNS and HTTPS can work even when the delivered page is protected. Verify the content available to an anonymous request, not only the hostname configuration.
Sources and methodology
Reviewed on 2026-10-03 against Vercel Deployment Protection documentation. This is an access-diagnosis workflow with illustrative route examples; no reader project settings or private deployment were inspected.
These references support the changeable facts and study findings discussed above. Results depend on each source's sample, date, market, query set, and measurement method.