I am attempting to index all of our items for a sitemap. I am using the Spatie Sitemap Generator package to do this. Currently we have over 65,000 items which is problematic because a sitemap can only contain 50,000 links. No matter though, the package has a solution: a Sitemap Index that points to multiple sitemaps. Because the items are not accessible from any other pages on the site I cannot just use the crawling feature and I must add links manually. So that is what I am doing below. However I have two problems:
- I get segmentation faults half the time
- I get this error when trying to write the sitemaps to a file:
Call to undefined method Spatie\Sitemap\Sitemap::getType() (View: /Users/noahgary/Projects/han-user-portal/vendor/spatie/laravel-sitemap/resources/views/sitemapIndex/index.blade.php)
I'm not exactly sure why I am getting this error or how to assure there is a type to get when this code is executed... But I know the problem code is here in the package:
<?= '<'.'?'.'xml version="1.0" encoding="UTF-8"?>'."\n" ?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
@foreach($tags as $tag)
@include('laravel-sitemap::sitemapIndex/' . $tag->getType())
@endforeach
</sitemapindex>
Here is my code:
$this->info('Generate Sitemap');
$this->info(config('app.url'));
$sitemapPath = public_path('sitemap.xml');
$siteMapIndex = SitemapIndex::create();
$page = 1;
$size = 1000;
$max_result_window = 10000;
$data = json_decode('{"sort":"updated_at","order":"DESC","page":'.$page.',"size":'.$size.',"active":1,"search_criteria":[{"field":"active","value":1,"operator":"eq","nested":false}], "source": "sitemap"}', true);
// Get first result
$response = $this->itemService->getByCriteria($data);
try {
$total_results = 0;
$this->info("Max result window: " . $max_result_window);
for ($i = 0; $i < 6; $i++)
// TODO: get total number of items for this loop
{
$sitemap = Sitemap::create(config('app.url'));
$this->info("Indexing items " . ($i*$max_result_window+1) . " to " . ($i+1)*$max_result_window);
while ($total_results < ($max_result_window - $size))
// While we have less than the maximum number of results allowed by elastic...
{
foreach ($response->hits->hits as $item) // Add a link for each item to the sitemap
{
$sitemap->add(Url::create("/shop/" . $item->_id));
}
// Some stats
$total_results = ($page * $size - ($size - sizeof($response->hits->hits)));
$this->info("Items indexed: " . ($max_result_window * $i + $total_results));
//Get next page
$page++;
$data = json_decode('{"sort":"updated_at","order":"DESC","from":' . ($i*$max_result_window) . ',"page":' . $page . ',"size":' . $size . ',"active":1,"search_criteria":[{"field":"active","value":1,"operator":"eq","nested":false}], "source": "sitemap"}', true);
$response = $this->itemService->getByCriteria($data);
}
//Reset. We are moving the result window.
$page = 1;
$total_results = 0;
// Write sitemap to sitemap index and move on to create a new index.
$siteMapIndex->add($sitemap);
$sitemap = null;
unset($sitemap);
}
} catch (\Exception $e) {
var_dump($response);
}
$siteMapIndex->writeToFile($sitemapPath);