RobotsTxtHandler project – How it works (part 3)


  1. Create new “Class Library” project.
  2. Add these NuGet packages to your project: (Please consider you need to have Feed Url of EPiserver NuGet server -> Link):EPiServer.CMS.UI.Core
  3. Entry  point: We need to define a “Normal MVC Controller” and decorate that with “GuiPlugIn” attribute:
            Area = EPiServer.PlugIn.PlugInArea.AdminMenu,
            DisplayName = "Robots.txt Handler",
            Description = "Robots.txt Handler",
            Url = Const.ModuleUrlBase + Const.Separator + Const.ModuleAdminController + Const.Separator + Const.ModuleAdminIndexAction,
            RequiredAccess = AccessLevel.Administer)]
        public class AdminController : Controller

    As you can see within attribute properties you can define the name and access level and base Url to use the module to route the URL correctly  into controller. Another important bit is “Area” property! This tells Episerver WHERE to show the  link of the plugin.  You can see full list and description here

  4. When a user clicks on plugin link in “AdminMenu” we want to show a list of available “Robot.txt” handler for each site to a user. To achieve this let’s add “Index”:
           public ActionResult Index()
                var robotTxts = robotsTxtRepository.All().ToList();
                var model = new AdminViewModel
                    Sites = GetSitesList(robotTxts.Select(a => a.SiteId)),
                    AvailableRobotTxts = robotTxts.Select(a => new AvailableRobotTxt { Id = a.Id.ExternalId, Name = siteList.Single(s => s.Id.ToString() == a.SiteId).Name })
                return PluginPartialView(Const.ModuleAdminIndexAction, model);

    And more than obvious we need to define ViewModel and repository class. You can see them on GitHub

  5. We need to add “Index.cshtml” to the project. Let’s create folder “\View\Admin” and create “Index.cshtml” under “Admin”:
    @model AdminViewModel
    @{  Layout = "Layout.cshtml"; }
    @if (Model.AvailableRobotTxts != null && Model.AvailableRobotTxts.Any())
            @foreach (var robotTxt in Model.AvailableRobotTxts)
                    <span><a href="@Url.Content(string.Format("{0}/{1}/{2}/{3}", Const.ModuleUrlBase,Const.ModuleAdminController,Const.ModuleAdminEditAction, robotTxt.Id))">Edit</a></span>
                    <span><a href="@Url.Content(string.Format("{0}/{1}/{2}/{3}", Const.ModuleUrlBase, Const.ModuleAdminController, Const.ModuleAdminDeleteAction, robotTxt.Id))">Delete</a></span>
    @if (Model.Sites != null && Model.Sites.Any())
        using (Html.BeginForm())
            @Html.DropDownList("SelectedSite", Model.Sites)
            @Html.TextAreaFor(a => a.RobotText, new { @rows = 20 })
            <input type="submit" value="Submit" />


  6. Create “module.config”. This file instructs Episerver how route segment should work and register the plugin DLL:
    <?xml version="1.0" encoding="utf-8"?>
    <module loadFromBin="false" productName="Zanganeh RobotsTxtHandler" >
        <add assembly="Zanganeh.RobotsTxtHandler" />
        <route url="{moduleArea}/{controller}/{action}/{id}">
            <add key="moduleArea" value="Zanganeh.RobotsTxtHandler" />
            <add key="controller" value="" />
            <add key="action" value="" />
            <add key="id" value="" />
  7. Create web.config  under “View” folder, so Razor template works fine:
    <?xml version="1.0"?>
        <sectionGroup name="system.web.webPages.razor" type="System.Web.WebPages.Razor.Configuration.RazorWebSectionGroup, System.Web.WebPages.Razor, Version=, Culture=neutral, PublicKeyToken=31BF3856AD364E35">
          <section name="host" type="System.Web.WebPages.Razor.Configuration.HostSection, System.Web.WebPages.Razor, Version=, Culture=neutral, PublicKeyToken=31BF3856AD364E35" requirePermission="false" />
          <section name="pages" type="System.Web.WebPages.Razor.Configuration.RazorPagesSection, System.Web.WebPages.Razor, Version=, Culture=neutral, PublicKeyToken=31BF3856AD364E35" requirePermission="false" />
        <host factoryType="System.Web.Mvc.MvcWebRazorHostFactory, System.Web.Mvc, Version=, Culture=neutral, PublicKeyToken=31BF3856AD364E35" />
        <pages pageBaseType="System.Web.Mvc.WebViewPage">
            <add namespace="System.Web.Mvc" />
            <add namespace="System.Web.Mvc.Ajax" />
            <add namespace="System.Web.Mvc.Html" />
            <add namespace="System.Web.Routing" />
            <add namespace="EPiServer.Framework.Web.Mvc.Html" />
            <add namespace="Zanganeh.RobotsTxtHandler" />
            <add namespace="Zanganeh.RobotsTxtHandler.ViewModel" />
            <add assembly="Zanganeh.RobotsTxtHandler" />
  8. Create new “nuspec” file to instruct NuGet Package Manager to copy “View” properly into proper area:
    <?xml version="1.0" encoding="utf-8"?>
    <package xmlns="">
    	<metadata xmlns="">
    		<tags>Robots EPiServer</tags>
        <file src="module.config" target="content\modules\Zanganeh.RobotsTxtHandler\module.config" />
        <file src="Views\web.config" target="content\modules\Zanganeh.RobotsTxtHandler\Views\web.config" />
        <file src="Views\Admin\Index.cshtml" target="content\modules\Zanganeh.RobotsTxtHandler\Views\Admin\Index.cshtml" />


  9. I’m using “MyGet” (can take a look using link) to build my project and generate NuGet file and hosting purpose.

You can fork GitHub and use push it to your own MyGet to see how it works. Valdis Iljuconoks mentioned good point why not extending Link. I just started working on this project for learning purpose and one of our client was asking for this feature and because it is simple I just picked it up to try the Episerver Plugins.

RobotsTxtHandler project – How it works (part 2)

You need to have Episerver CMS project (CMS >= 10.0.2). Use Episerver NuGet (if you want check here) and search for Zanganeh.RobotsTxtHandler or directly running code below in “Package Manager Console”:

PM> Install-Package Zanganeh.RobotsTxtHandler

Then re-build  and run project. Then goto {siteurl}/EPiServer/CMS/Admin/Default.aspx and on left hand menu click on “tools -> Robots.txt Handler”:


You can select current site from “Dropdown”and enter text you want to show as “robots.txt” file for that site.

When you save you can see list:



And if you got to http://{siteurl}/robots.txt you can see what you entered in the textbox:



You can delete or edit robots.txt value using same list:


The  idea behind the scene is quite simple. For each site we store data of robots.txt into Dynamic Data Store. And having http handler that based on current site extract associated robots.txt and show the content. In next step I will describe Episerver add-on Gui.

RobotsTxtHandler project (part 1)

As I wanted to get some  experience with Episerver plugin I started this project. This package allow site admin to add/edit/delete “robots.txt” file for Episerver site(s). Requirements  are as below:

  1. Provide “/robots.txt” handler
  2. Support multi-site
  3. Support multilingual
  4. Support multi-channel (e.g. for mobile channel provides different robots.txt)
  5. Easy admin area
  6. Support “default” robots.txt
  7. For UAT site disallow robots to  crawl the site (using web.config to override all  existing configs)
  8. Give some basic analytics data
  9. Plugin is based on MVC (for my own learning purpose only!)

Based  on above I broke it down to into three phases

Phase 1:

  1. Provide “/robots.txt” handler
  2. Support multi-site
  3. Plugin is based on MVC (for my own learning purpose only!)

Phase 2:

  1. Easy admin area
  2. Support “default” robots.txt
  3. For UAT site disallow robots to  crawl the site (using web.config to override all  existing configs)

Phase 3:

  1. Give some basic analytics data
  2. Support multilingual
  3. Support multi-channel (e.g. for mobile channel provides different robots.txt)

I already release the first RC for Phase 1 and in this tutorial, I will try to explain the challenges and what I learned. You  can access the  repo via:

More than happy to get feedbacks on githib. Next step I will describe the architecture and base of plugin, nuget  package and MyGet integration!