-1

I am attempting to index all of our items for a sitemap. I am using the Spatie Sitemap Generator package to do this. Currently we have over 65,000 items which is problematic because a sitemap can only contain 50,000 links. No matter though, the package has a solution: a Sitemap Index that points to multiple sitemaps. Because the items are not accessible from any other pages on the site I cannot just use the crawling feature and I must add links manually. So that is what I am doing below. However I have two problems:

  1. I get segmentation faults half the time
  2. I get this error when trying to write the sitemaps to a file:
Call to undefined method Spatie\Sitemap\Sitemap::getType() (View: /Users/noahgary/Projects/han-user-portal/vendor/spatie/laravel-sitemap/resources/views/sitemapIndex/index.blade.php)

I'm not exactly sure why I am getting this error or how to assure there is a type to get when this code is executed... But I know the problem code is here in the package:

<?= '<'.'?'.'xml version="1.0" encoding="UTF-8"?>'."\n" ?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
@foreach($tags as $tag)
    @include('laravel-sitemap::sitemapIndex/' . $tag->getType())
@endforeach
</sitemapindex>

Here is my code:

        $this->info('Generate Sitemap');
        $this->info(config('app.url'));

        $sitemapPath = public_path('sitemap.xml');
        $siteMapIndex = SitemapIndex::create();

        $page = 1;
        $size = 1000;
        $max_result_window = 10000;
        $data = json_decode('{"sort":"updated_at","order":"DESC","page":'.$page.',"size":'.$size.',"active":1,"search_criteria":[{"field":"active","value":1,"operator":"eq","nested":false}], "source": "sitemap"}', true);

        // Get first result
        $response = $this->itemService->getByCriteria($data);

        try {
            $total_results = 0;

            $this->info("Max result window: " . $max_result_window);

            for ($i = 0; $i < 6; $i++)
            // TODO: get total number of items for this loop
            {
                $sitemap = Sitemap::create(config('app.url'));

                $this->info("Indexing items " . ($i*$max_result_window+1) . " to " . ($i+1)*$max_result_window);
                while ($total_results < ($max_result_window - $size))
                    // While we have less than the maximum number of results allowed by elastic...
                {
                    foreach ($response->hits->hits as $item) // Add a link for each item to the sitemap
                    {
                        $sitemap->add(Url::create("/shop/" . $item->_id));
                    }

                    // Some stats

                    $total_results = ($page * $size - ($size - sizeof($response->hits->hits)));

                    $this->info("Items indexed: " . ($max_result_window * $i + $total_results));

                    //Get next page
                    $page++;
                    $data = json_decode('{"sort":"updated_at","order":"DESC","from":' . ($i*$max_result_window) . ',"page":' . $page . ',"size":' . $size . ',"active":1,"search_criteria":[{"field":"active","value":1,"operator":"eq","nested":false}], "source": "sitemap"}', true);
                    $response = $this->itemService->getByCriteria($data);
                }
                //Reset. We are moving the result window.
                $page = 1;
                $total_results = 0;

                // Write sitemap to sitemap index and move on to create a new index.
                $siteMapIndex->add($sitemap);
                $sitemap = null;
                unset($sitemap);
            }
        } catch (\Exception $e) {
            var_dump($response);
        }

        $siteMapIndex->writeToFile($sitemapPath);
apokryfos
  • 38,771
  • 9
  • 70
  • 114
Noah Gary
  • 916
  • 12
  • 25

1 Answers1

0

Okay, So the reason I got this error was because I was trying to add a sitemap object to the index. I should have added it like so:

$sitemap->writeToFile($sitemapPath);
$siteMapIndex->add($sitemapPath);

The documentation gives an example of adding a sitemap object but this is ONLY in the case of changing the lastModificationDate like the example below from the documentation:

SitemapIndex::create()
    ->add('/pages_sitemap.xml')
    ->add(Sitemap::create('/posts_sitemap.xml')
        ->setLastModificationDate(Carbon::yesterday()))
    ->writeToFile($sitemapIndexPath);

So, the correct implementation is:

  1. Create sitemap
  2. Add links
  3. Write sitemap
  4. Add sitemap, by its path, to the sitemap index
Noah Gary
  • 916
  • 12
  • 25