You need to have Episerver CMS project (CMS >= 10.0.2). Use Episerver NuGet (if you want check here) and search for Zanganeh.RobotsTxtHandler or directly running code below in “Package Manager Console”:
PM> Install-Package Zanganeh.RobotsTxtHandler
Then re-build and run project. Then goto {siteurl}/EPiServer/CMS/Admin/Default.aspx and on left hand menu click on “tools -> Robots.txt Handler”:
You can select current site from “Dropdown”and enter text you want to show as “robots.txt” file for that site.
When you save you can see list:
And if you got to http://{siteurl}/robots.txt you can see what you entered in the textbox:
You can delete or edit robots.txt value using same list:
The idea behind the scene is quite simple. For each site we store data of robots.txt into Dynamic Data Store. And having http handler that based on current site extract associated robots.txt and show the content. In next step I will describe Episerver add-on Gui.
Valdis Iljuconoks
November 16, 2016 — 10:54 pm
Hi,
Nice to see attempts to solve this SEO issue. I’m just wondering why you didn’t contribute to existing robots handler?
http://nuget.episerver.com/en/OtherPages/Package/?packageId=POSSIBLE.RobotsTxtHandler
zanganeh
November 17, 2016 — 8:41 am
Hi Valdis,
My main aim was to learn plugin concept for episerver in MVC and using Razor. I took a look at the other projects and seems like most of them are not MVC and POSSIBLE is MVC but not using Razor! All of the are quite well written and I just done this for learning purpose only to be honest 🙂 Thanks for sharing your concern with me!
Regards,
Aria