# Search: crawl instead of search markdown content files

**URL:** <https://forum.getkirby.com/t/search-crawl-instead-of-search-markdown-content-files/8756>\
**Category:** Questions\
**Tags:** v2\
**Created:** [October 11, 2017, 6:51am UTC](https://forum.getkirby.com/t/search-crawl-instead-of-search-markdown-content-files/8756 "2017-10-11T06:51:05Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![bvdputte](https://dub1.discourse-cdn.com/flex017/user_avatar/forum.getkirby.com/bvdputte/32/8179_2.png) [@bvdputte](https://forum.getkirby.com/u/bvdputte)\
**Post date:** [October 11, 2017, 6:51am UTC](https://forum.getkirby.com/t/search-crawl-instead-of-search-markdown-content-files/8756/1 "2017-10-11T06:51:05Z")

</div>

In some websites I create “hidden sections” or “hidden subpages” to manage little pieces of structured content that are not directly accessible, but used in another page. They are usually not directly accessible because they are “too small” to create a dedicated page out of them.

How can I work around the built-in search, which returns the page where the content is “stored” (_which is usually hidden/not directly accessible_) instead of “used”.

How do you work with this issue? I can imagine e.g. the [modules plugin](https://github.com/getkirby-plugins/modules-plugin) — _I don’t use it_— must suffer from this too. Is there some kind of “crawler” around that can be used for this purpose? Or is there a way to figure out where the content is used instead of stored in the search controller?

I hope this is making sense? If not, let me know, I’ll try to elaborate this.

---

<div class="post-metadata">

**Author:** ![jimbobrjames](https://dub1.discourse-cdn.com/flex017/user_avatar/forum.getkirby.com/jimbobrjames/32/5802_2.png) [@jimbobrjames](https://forum.getkirby.com/u/jimbobrjames)\
**Post date:** [October 11, 2017, 8:18am UTC](https://forum.getkirby.com/t/search-crawl-instead-of-search-markdown-content-files/8756/2 "2017-10-11T08:18:01Z")

</div>

Can you not just filter those pages out in the [search controller](https://getkirby.com/docs/cookbook/search)?

Not tested, but i imagine if you changed this line to include a filter, you can get what you want?

```auto
$results = $site->search($query, 'title|text');

```

I guess you would need something unique on those pages to filter on, or use something like [AutoID](https://github.com/helllicht/kirby-autoid) to feed the filter a string of page ID numbers to exclude.

---

<div class="post-metadata">

**Author:** ![texnixe](https://dub1.discourse-cdn.com/flex017/user_avatar/forum.getkirby.com/texnixe/32/5754_2.png) [@texnixe](https://forum.getkirby.com/u/texnixe)\
**Post date:** [October 11, 2017, 9:06am UTC](https://forum.getkirby.com/t/search-crawl-instead-of-search-markdown-content-files/8756/3 "2017-10-11T09:06:48Z")

</div>

If you filter them out, they are no included in the search, so that is not really that useful.

I think you would need a custom method and a relationship field that tells Kirby where that content appears (as Kirby would not be able to find that out by itself without parsing your templates/controllers). Then you could output that link In your search results instead of the page URL.

I am not sure how third party extensions like the Algolia plugin work, maybe that could be an alternative?

---

<div class="post-metadata">

**Author:** ![jimbobrjames](https://dub1.discourse-cdn.com/flex017/user_avatar/forum.getkirby.com/jimbobrjames/32/5802_2.png) [@jimbobrjames](https://forum.getkirby.com/u/jimbobrjames)\
**Post date:** [October 11, 2017, 9:39am UTC](https://forum.getkirby.com/t/search-crawl-instead-of-search-markdown-content-files/8756/4 "2017-10-11T09:39:09Z")

</div>

> [@texnixe](#):
>
> If you filter them out, they are no included in the search, so that is not really that useful.

But that is what @bvdputte wants. These are not whole pages, but utility pages used as part of other pages, and not meant to be viewed directly, and should not appear in the search.

---

<div class="post-metadata">

**Author:** ![texnixe](https://dub1.discourse-cdn.com/flex017/user_avatar/forum.getkirby.com/texnixe/32/5754_2.png) [@texnixe](https://forum.getkirby.com/u/texnixe)\
**Post date:** [October 11, 2017, 10:01am UTC](https://forum.getkirby.com/t/search-crawl-instead-of-search-markdown-content-files/8756/5 "2017-10-11T10:01:47Z")

</div>

> [@jimbobrjames](#):
>
> These are not whole pages, but utility pages used as part of other pages, and not meant to be viewed directly, and should not appear in the search.

I answer with this quote:

> [@bvdputte](#):
>
> Or is there a way to figure out where the content is used instead of stored in the search controller?

So for me that means, @bvdputte does want the content to appear in the search results, but not with their own (inaccessible) URL, but with the URL of the page where the content appears. Correct me if I’m wrong, @bvdputte.

---

<div class="post-metadata">

**Author:** ![bvdputte](https://dub1.discourse-cdn.com/flex017/user_avatar/forum.getkirby.com/bvdputte/32/8179_2.png) [@bvdputte](https://forum.getkirby.com/u/bvdputte)\
**Post date:** [October 11, 2017, 10:03am UTC](https://forum.getkirby.com/t/search-crawl-instead-of-search-markdown-content-files/8756/6 "2017-10-11T10:03:34Z")

</div>

> [@texnixe](#):
>
> So for me that means, @bvdputte does want the content to appear in the search results, but not with their own (inaccessible) URL, but with the URL of the page where the content appears.

You are 100% right. This is what I would like to achieve.

---

<div class="post-metadata">

**Author:** ![jimbobrjames](https://dub1.discourse-cdn.com/flex017/user_avatar/forum.getkirby.com/jimbobrjames/32/5802_2.png) [@jimbobrjames](https://forum.getkirby.com/u/jimbobrjames)\
**Post date:** [October 11, 2017, 10:07am UTC](https://forum.getkirby.com/t/search-crawl-instead-of-search-markdown-content-files/8756/7 "2017-10-11T10:07:53Z")

</div>

Ok so, help me understand how the search works then…

Consider that Page D is constructed from data contained in fields on pages A, B and C. If you filter out pages A, B and C, does that mean that Kirby will not see this data as belonging to Page D, and it will not be searchable? Page D will essentially be empty as far as the search is concerned?

---

<div class="post-metadata">

**Author:** ![bvdputte](https://dub1.discourse-cdn.com/flex017/user_avatar/forum.getkirby.com/bvdputte/32/8179_2.png) [@bvdputte](https://forum.getkirby.com/u/bvdputte)\
**Post date:** [October 11, 2017, 10:11am UTC](https://forum.getkirby.com/t/search-crawl-instead-of-search-markdown-content-files/8756/8 "2017-10-11T10:11:22Z")

</div>

> [@jimbobrjames](#):
>
> Consider that Page D is constructed from data contained in fields on pages A, B and C. If you filter out pages A, B and C, does that mean that Kirby will not see this data as belonging to Page D, and it will not be searchable? Page D will essentially be empty as far as the search is concerned?

Exactly. As you’ld see in the `page-d.en.md` file in the `/content` folder 😉  
And this is where the Kirby search function looks into for matches.

The logic that _aggregates_ those 3 files into 1 lives in the template, and Kirby does not know this of _relationship_ (as @texnixe beautifully said) outside that template.

---

<div class="post-metadata">

**Author:** ![jimbobrjames](https://dub1.discourse-cdn.com/flex017/user_avatar/forum.getkirby.com/jimbobrjames/32/5802_2.png) [@jimbobrjames](https://forum.getkirby.com/u/jimbobrjames)\
**Post date:** [October 11, 2017, 10:14am UTC](https://forum.getkirby.com/t/search-crawl-instead-of-search-markdown-content-files/8756/9 "2017-10-11T10:14:48Z")

</div>

I see. So maybe this [plugin will help then](https://github.com/flokosiol/kirby-redirect)? You essentially want to redirect direct hits to pages A, B and C to page D. Would that work for you?

You do have another problem - getting Google and other search engines to see the “real” page in search results. You will probably have to get clever with the site map.

---

<div class="post-metadata">

**Author:** ![texnixe](https://dub1.discourse-cdn.com/flex017/user_avatar/forum.getkirby.com/texnixe/32/5754_2.png) [@texnixe](https://forum.getkirby.com/u/texnixe)\
**Post date:** [October 11, 2017, 10:21am UTC](https://forum.getkirby.com/t/search-crawl-instead-of-search-markdown-content-files/8756/10 "2017-10-11T10:21:53Z")

</div>

> [@jimbobrjames](#):
>
> You do have another problem - getting Google and other search engines to see the “real” page in search results.

Why? Google crawls URLs and their content, I don’t see how that would affect search results.

---

<div class="post-metadata">

**Author:** ![jimbobrjames](https://dub1.discourse-cdn.com/flex017/user_avatar/forum.getkirby.com/jimbobrjames/32/5802_2.png) [@jimbobrjames](https://forum.getkirby.com/u/jimbobrjames)\
**Post date:** [October 11, 2017, 10:23am UTC](https://forum.getkirby.com/t/search-crawl-instead-of-search-markdown-content-files/8756/11 "2017-10-11T10:23:02Z")

</div>

> [@texnixe](#):
>
> Why? Google crawls URLs and their content

Because they will index pages A, B and C, when all is is needed is Page D.

---

<div class="post-metadata">

**Author:** ![bvdputte](https://dub1.discourse-cdn.com/flex017/user_avatar/forum.getkirby.com/bvdputte/32/8179_2.png) [@bvdputte](https://forum.getkirby.com/u/bvdputte)\
**Post date:** [October 11, 2017, 10:23am UTC](https://forum.getkirby.com/t/search-crawl-instead-of-search-markdown-content-files/8756/12 "2017-10-11T10:23:26Z")

</div>

> [@texnixe](#):
>
> Google crawls URLs and their content, I don’t see how that would affect search results.

Exactly. Google indexes URL’s. And this is what makes it ambiguous for users of the website who use the “on-site search” → this yields different results then google. Which is (technically) obvious for above mentioned reasons, but not wanted/expected behaviour.

---

<div class="post-metadata">

**Author:** ![texnixe](https://dub1.discourse-cdn.com/flex017/user_avatar/forum.getkirby.com/texnixe/32/5754_2.png) [@texnixe](https://forum.getkirby.com/u/texnixe)\
**Post date:** [October 11, 2017, 10:23am UTC](https://forum.getkirby.com/t/search-crawl-instead-of-search-markdown-content-files/8756/13 "2017-10-11T10:23:47Z")

</div>

No, if there is no link to those A, B, C pages, because they are not accessible, no problem.

---

<div class="post-metadata">

**Author:** ![texnixe](https://dub1.discourse-cdn.com/flex017/user_avatar/forum.getkirby.com/texnixe/32/5754_2.png) [@texnixe](https://forum.getkirby.com/u/texnixe)\
**Post date:** [October 11, 2017, 10:27am UTC](https://forum.getkirby.com/t/search-crawl-instead-of-search-markdown-content-files/8756/14 "2017-10-11T10:27:14Z")

</div>

> [@bvdputte](#):
>
> And this is what makes it ambiguous for users of the website who use the “on-site search” -\> this yields different results then google.

That’s true. The built-in search method is just useful in very simple cases, where basically a folder is a page that is accessible via its URL. It is a page/site method after all, and pages are tied to folders.

That’s why you have to make sure that those folders do not appear in URLs (using routes, for example) and that they don’t appear in the search results as pages, but at the same time make them appear in search results for those aggregated pages (using relationships or some other search method).

---

<div class="post-metadata">

**Author:** ![jimbobrjames](https://dub1.discourse-cdn.com/flex017/user_avatar/forum.getkirby.com/jimbobrjames/32/5802_2.png) [@jimbobrjames](https://forum.getkirby.com/u/jimbobrjames)\
**Post date:** [October 11, 2017, 10:27am UTC](https://forum.getkirby.com/t/search-crawl-instead-of-search-markdown-content-files/8756/15 "2017-10-11T10:27:38Z")

</div>

I have had pages like this show up in search results accidentally because they have found their way somehow into the sitemap.xml file. It took some time to flush them out of Googles index.

---

<div class="post-metadata">

**Author:** ![bvdputte](https://dub1.discourse-cdn.com/flex017/user_avatar/forum.getkirby.com/bvdputte/32/8179_2.png) [@bvdputte](https://forum.getkirby.com/u/bvdputte)\
**Post date:** [October 12, 2017, 6:19pm UTC](https://forum.getkirby.com/t/search-crawl-instead-of-search-markdown-content-files/8756/16 "2017-10-12T18:19:26Z")

</div>

So, now we have identified the symptoms, are there people with a solution/workaround?  
Tbh I’m a bit surprised no one has ever experienced this? 👀

---

<div class="post-metadata">

**Author:** ![texnixe](https://dub1.discourse-cdn.com/flex017/user_avatar/forum.getkirby.com/texnixe/32/5754_2.png) [@texnixe](https://forum.getkirby.com/u/texnixe)\
**Post date:** [October 12, 2017, 10:33pm UTC](https://forum.getkirby.com/t/search-crawl-instead-of-search-markdown-content-files/8756/17 "2017-10-12T22:33:55Z")

</div>

Just some ideas:

- [https://codecanyon.net/item/easy-web-search-php-search-engine-with-image-search-and-crawling-system/17574164](https://codecanyon.net/item/easy-web-search-php-search-engine-with-image-search-and-crawling-system/17574164)

- use curl and feed it your (cleaned up) sitemap (probably not very performant unless cached or indexed in a database), really basic example, just as a starting point:

```auto
<?php

  function getData($url) {
  	$ch = curl_init();
  	$timeout = 5;
  	curl_setopt($ch, CURLOPT_URL, $url);
  	curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  	curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
  	$data = curl_exec($ch);
  	curl_close($ch);
  	return $data;
  }

  $q = 'Kirby';
  $matches = [];
  foreach($site->index() as $p) {
    $returned_content = getData($p->url());
    if (strpos($returned_content, $q) !== false) {
        $matches[] = $p->url();
    }
  }
  dump($matches);

?>

```

- maybe even read the cache files, provided the cache is rebuilt after every flush.

- or, as I suggested above, custom search with relationships

---

<div class="post-metadata">

**Author:** ![bvdputte](https://dub1.discourse-cdn.com/flex017/user_avatar/forum.getkirby.com/bvdputte/32/8179_2.png) [@bvdputte](https://forum.getkirby.com/u/bvdputte)\
**Post date:** [October 13, 2017, 6:04am UTC](https://forum.getkirby.com/t/search-crawl-instead-of-search-markdown-content-files/8756/18 "2017-10-13T06:04:12Z")

</div>

> [@texnixe](#):
>
> custom search with relationships

Relationships is quite another beast to handle imho. I know e.g. ola has developed a relationship plugin, but [as I expressed my concern](https://forum.getkirby.com/t/relationship-a-sortable-multiselect-field/8221), it doesn’t handle “keep established relationships working” well (e.g. in case of url renames or page deletions)? You’re on your own for accounting this, via hooks and other plugins 🙂. This makes the custom development dependency graph quite huge for a _little use-case_; imho it feels like building a _house of cards_.

Thanks for your insights in this, @texnixe.

When I find some time, I might have a go at your other proposal about building a “crawler” ourselves and searching its built cache then. Performance will be a pickle I’m afraid. Maybe cron can help with that? Does anyone know of good, simple, open source PHP crawler that we could plug in for this? I would like to avoid re-inventing the wheel.

---

<div class="post-metadata">

**Author:** ![texnixe](https://dub1.discourse-cdn.com/flex017/user_avatar/forum.getkirby.com/texnixe/32/5754_2.png) [@texnixe](https://forum.getkirby.com/u/texnixe)\
**Post date:** [October 13, 2017, 7:57am UTC](https://forum.getkirby.com/t/search-crawl-instead-of-search-markdown-content-files/8756/19 "2017-10-13T07:57:35Z")

</div>

Well, maybe someone comes up with some better ideas.

I see the problem with relationships, but that is something you will have to take care of no matter which of the methods you use. With a solution like Algolia you have to keep your index intact, with a database index as well, although maybe easier to achieve.

I don’t know your exact use case, but I would think that if you tell a non-accessible module page that it is part of container X (rather than telling X what modules belong to it) and if X is less likely to change its URL or be deleted, then this relationship thingy could work.

But maybe the crawling solutions are more reliable after all. Anyway, I was just trying to come up with some ideas, seeing that up to now nobody has come around with a ready-to-use solution. I have never tested any of these approaches myself.

Anyway, interesting topic. Let me finish this with this list of crawlers: [https://github.com/BruceDone/awesome-crawler](https://github.com/BruceDone/awesome-crawler)

---

<div class="post-metadata">

**Author:** ![pedroborges](https://dub1.discourse-cdn.com/flex017/user_avatar/forum.getkirby.com/pedroborges/32/7038_2.png) [@pedroborges](https://forum.getkirby.com/u/pedroborges)\
**Post date:** [October 13, 2017, 9:12am UTC](https://forum.getkirby.com/t/search-crawl-instead-of-search-markdown-content-files/8756/20 "2017-10-13T09:12:21Z")

</div>

> > So for me that means, @bvdputte does want the content to appear in the search results, but not with their own (inaccessible) URL, but with the URL of the page where the content appears.
> 
> You are 100% right. This is what I would like to achieve.

I’m not sure if this solution would work on your search context, but in a different situation I have used [Kirby models](https://getkirby.com/docs/developer-guide/advanced/models) to change the URL of related pages that shouldn’t exist on their own.

```php
<?php // site/models/{related-page-template-name}.php

class Relatedpage extends Page
{
      public function url()
      {
          return $this->parent()->url();
      }
}

```

[Next page](https://forum.getkirby.com/t/search-crawl-instead-of-search-markdown-content-files/8756.md?page=2)