[Help wanted] 🔗 Broken Links Checker Widget

wottpal · November 14, 2017, 12:02am

Hey,
I’ve written a little panel widget to scan through all pages and determine if links are broken (checking if a 404 is returned by the URL). And so far it seems to work, I’ve even added support for excluding pages and including other fields than ‘text’ just a few moments ago: https://github.com/wottpal/kirby-broken-links-widget

What I think is really missing to make this plugin bulletproof is making this work “async”. So maybe a bit like ImageKit by @fabianmichael with a start-button and a little progress bar. But I have no clue where to start… I tried to dig through ImageKits source which became quite overwhelming very soon so maybe anybody can sketch out a quick solution to my problem.

Thanks!
Dennis

anon77445132 · November 14, 2017, 7:59am

Thanks,

If I have something like (link: /error text: Error-page popup: yes) in my text, I get a broken link message. Something is wrong…

Kirby 2.5.7

texnixe · November 14, 2017, 8:19am

@anon77445132 Haha, good joke. The error page does of course return a 404 so that is not surprising. But who would want to link to the error page.

Maybe you should exclude it if you want to send your users to the error page without a broken link:

anon77445132 · November 14, 2017, 8:59am

For the editors of Sportanglerverein Barchfeld e.V., I have added some internal pages about e.g. the use of the panel. On an extended copy of Kirby CMS Clientmanual, I have added this link.

To exclude this link is a very good idea. Sonja, thank you very much for your hint.

But that is strange from my point of view.
The link to this page is not broken (I can reach that page directly), so for me it is wrong to show it in such a report!

texnixe · November 14, 2017, 9:09am

I think this is an edge case. Exclude your internal pages and you are all set.

anon77445132 · November 14, 2017, 9:19am

@wottpal:

At the moment it seems that I cannot exclude that link (/error or error), I can only exclude a whole page from scanning…

wottpal · November 14, 2017, 12:05pm

Hey @anon77445132,
now in v0.3.0 you can exclude specific page-ids or absolute external links (if enabled). I even added /error as a default value. Please confirm if it’s working for you!

PS.: Still help needed!

fabianmichael · November 14, 2017, 12:50pm

To make things async, ImageKit defers possible expensive tasks after the panel has loaded. I.e. Instead of scanning the whole thumbs folder to generate the generated/pending thumbnail counts when the widget HTML is generated, these statistics are lazyloaded via AJAX.

To make things async, you have to provide an API for you widget, which is accessible via JavaScript. Kirby’s router feature is your friend here, but you need to handle things like authentification and i18n yourself.

API Code with Authentification:

github.com

fabianmichael/kirby-imagekit/blob/master/widgets/imagekit/lib/api.php

<?php

namespace Kirby\Plugins\ImageKit\Widget;

use Response;
use Exception;

use Kirby\Plugins\ImageKit\LazyThumb;
use Kirby\Plugins\ImageKit\ComplainingThumb;

use Whoops\Handler\Handler;
use Whoops\Handler\CallbackHandler;


class API {
  
  public $kirby;
  
  public static function instance() {
    static $instance;

This file has been truncated. show original

The actual crawler component that scans pages:

github.com

fabianmichael/kirby-imagekit/blob/master/widgets/imagekit/lib/apicrawlerresponse.php

<?php

namespace Kirby\Plugins\ImageKit\Widget;

use Exception;
use DOMDocument;
use Kirby;
use Response;
use Kirby\Plugins\ImageKit\LazyThumb;
use Url;
use V;

class APICrawlerResponse extends \Kirby\Component\Response {

  public function __construct(Kirby $kirby) {
    parent::__construct($kirby);

    // Register listeners for redirects
    header_register_callback([$this,'detectRedirectRequest']);
    register_shutdown_function([$this,'detectRedirectRequest']);

This file has been truncated. show original

When ImageKit scans the whole site for thumbnails, it does do so by first fetching a sitemap via the API. The sitemap is iterated over to generate an HTTP request to every single page for triggering thumbnail job creation. The JavaScript API sends a custom header (X-ImageKit-Indexing: 1) to tell the server: Generate the page like you would normally do, but after that return me a JSON result instead of the whole HTML page.

ImageKit is somewhat smart here, as every indexing request also searches the page for rel="prev" and rel="next" links for making paginated pages crawlable as well (this happens on the server). If such links are found, they are added to the API response object and added to the scanning queue (JavaScript), if they’re not already in there.

github.com

fabianmichael/kirby-imagekit/blob/master/widgets/imagekit/assets/js/src/widget.js#L113


      } else {
        stop(ACTION_CREATE);
        complete(response.data);
      }
    });
  }
  
  doCreate();
}


function index(step, complete, error) {
  reset();
  start(ACTION_INDEX);
      
  step      = step || function(){};
  complete  = complete || function(){};
  error     = error || function(){};
  
  api(ACTION_INDEX, function (response) {
    var i = 0;
    var pageUrls = response.data;

I hope, I could help you a bit?

wottpal · November 14, 2017, 10:06pm

Thanks a lot Fabian, I’ll definitely look into this! I’m also thinking about not putting to much effort into this plugin and maybe concentrate on a nice Kirby-3 version

Also for reference, I opened an issue about that (https://github.com/wottpal/kirby-broken-links-widget/issues/1) and if anybody is up to implementing this a PR would be warmly welcome.