Robots.txt
A text file at the site root telling search crawlers which paths they may and may not crawl.
In Arabic ملف robots.txt
What is Robots.txt?
Robots.txt is a text file at the site root, such as example.com/robots.txt, that crawlers read before starting. Simple rules like Disallow set which paths they skip, such as cart, account and internal search pages, and it can point to the sitemap URL. Most platforms and WordPress plugins create it automatically.
The file controls crawling and does not necessarily prevent indexing. A page blocked in it can still appear in results without a description if other sites link to it. And because the file is public for anyone to read, it is no way to hide sensitive pages; it can even point to them. Real protection means a password or removing the page.
Why does Robots.txt matter in marketing?
One wrong line in this file can block your entire site from Google, so it deserves a review after every change.
- 01
Steering crawling to key pages
Blocking cart, account and internal search pages leaves Google's time for products, categories and content.
- 02
A quick post-launch check
The first thing to check after a migration or launch is this file, because staging sites are usually fully blocked.
- 03
Diagnosing a sudden indexing drop
When the number of indexed pages falls suddenly, checking this file is one of the first steps to finding the cause.
Worked example
- 1Situation
A WooCommerce store in Jazan wants to stop crawlers from crawling its cart and checkout pages.
- 2Data
The developer adds two Disallow rules for those paths, but by mistake leaves a line that blocks the whole site.
- 3Calculation
Indexed pages in Search Console drop from 400 to 40, a fall of 360 ÷ 400 = 90%.
- 4Decision
The store removes the full-block line, keeps the two rules, tests the file and watches crawling recover gradually.
Common mistakes
Leaving Disallow: / after launch
This line blocks the whole site and is often left over from development. Open the file on launch day and confirm it is gone.
Blocking a page that has noindex
If the page is blocked in the file, Google never sees its noindex tag and it may stay indexed. Allow crawling so the tag can be read.
Using it to hide private paths
The file is public, and listing your admin panel or customer files there reveals them. Protect those paths with a login.
Frequently asked questions
Where do I find my robots.txt and how do I test it?
Type your domain followed by /robots.txt in a browser to view it. In Search Console, the robots.txt report shows the version Google read and any errors. To test a specific URL, use URL Inspection to see whether crawling it is allowed.
What is the difference between robots.txt and noindex?
Robots.txt blocks crawling, meaning visiting the page, but does not guarantee it stays out of results. Noindex allows the visit and asks not to index it, so the page drops from results. To remove a page from search, use noindex; to save crawling on worthless pages, use robots.txt.
Takteek services that help here
We check robots.txt on the first day after any launch or migration, and make sure its rules block neither design files nor pages carrying noindex.
Talk to Takteek
Want to put this to work for your business?
Tell us about your business in four steps and we will come back with a clear plan that starts from your current numbers. The consultation is free.
- A plan built on your numbers
- One team for ads, content and store
- Free consultation, no commitment


