If you use a query string instead of the page parameter, only the first page gets cached, all other pages are then pulled dynamically.
$albums = $page->children()->paginate([
'limit' => 2,
'method' => 'query',
'variable' => 'p'
]);
So this could be a workaround to at least cache the first page of results.