Run multiple Magento DataFlow profiles in sequence

Question

I am developing a Magento extension that allows users to import categories, products and other kinds of data from a different web shop solution into Magento. In order to accomplish this, I have set up several DataFlow advanced profiles that take care of the imports.

So far it's working fine, but the profiles need to be run in the appropriate order (first categories, then products, etc.). To make it as easy as possible for non-technical users and also to minimize the chances of human error, I would like to automatically run the profiles in sequence. Ideally you would just press one "Run profile" button and then watch the grass growing for a few hours while DataFlow handles the profiles.

I cannot seem to accomplish this. My main question is - is this possible? And if yes, then how?

I have tried to combine multiple profiles in the same XML file (literally just copy-pasting 2 profiles in one XML) but this didn't work. The input files were parsed, but somehow the Adapter classes that were responsible for the actual import weren't being run.

Is there maybe a way to tell a profile to begin another profile when it's done (through the "finish" method for example)? Or perhaps I should write a small control panel where you can click on the "Run" button and then the control panel handles the sequence through some AJAX voodoo?

Using a command-line script is not an option, unfortunately, and neither is using Magmi.

For completeness, I must mention that all the profiles consist of:

• IO adapter that reads a CSV file
• Custom CSV parser (copied from the default DataFlow CSV parser with a couple of tweaks to account for quirks in the input file)
• Custom adapter that does the importing

Answer 1

This is how I run a single profile as part of a bigger cron job. It should be easy to make it run several profiles. And you should be able to make it run from pressing a button. This script fakes an admin user being logged into the backend and pressing some buttons. It looks a bit complicated, but this was the only working method after some others did not work properly because the script was running as a cron job.

Place both script files in the shell folder in magento's root dir:

mag_import.php:

<?php

/**
 * Path to the root of your magento installation
 */
$root = '/absolute/path/to/your/magento/root/';

/**
 * Url to your magento shop.
 */
$url = 'http://www.mygreatwebshop.url/';

/**
 * relative path from the magento root to the login file.
 */
$login = 'shell/mag_login.php';

/**
 * name of the logfile, will be places in magentoroot/var/log/
 */
$logFileName = 'import.log';

/**
 * how many products will be parsed at each post. Usually 10-50.
 */
$atOnce = 25;

/**
 * Dataflow profile id
 */    
$profileId = 8;

/**
 * DO NOT EDIT BELOW THIS LINE
 */
function convert($size) {
    $unit = array('b', 'kb', 'mb', 'gb', 'tb', 'pb');
    return @round($size / pow(1024, ($i = floor(log($size, 1024)))), 2) . ' ' . $unit[$i];
}

set_time_limit(0);

if (!isset($profileId)) {
    exit("\nPlease specify a profile id. You can find it in the admin panel->Import/Export->Profiles.\nUsage: \n\t\t php -f $argv[0] PROFILE_ID\n\t example: php -f $argv[0] 7\n");
}

$recordCount = 0;

require_once $root . 'app/Mage.php';
ob_implicit_flush();
Mage::app()->setCurrentStore(Mage_Core_Model_App::ADMIN_STORE_ID);

//starting the import
Mage::log("\n\n", null, $logFileName);
Mage::log(convert(memory_get_usage()) . " - " . "STARTING IMPORT", null, $logFileName);

$profile = Mage::getModel('dataflow/profile');
$userModel = Mage::getModel('admin/user');
$userModel->setUserId(0);

Mage::getSingleton('admin/session')->setUser($userModel);

if ($profileId) {
    $profile->load($profileId);
    if (!$profile->getId()) {
        Mage::getSingleton('adminhtml/session')->addError('ERROR: Could not load profile');
    }
}

/**
 * get the login information.
 */
exec("/usr/bin/php -f {$root}{$login}", $result);

$loginInformation = json_decode($result[0]);
$sessionId = $loginInformation->sessionId;
$formKey = $loginInformation->formKey;

//clean dataflow_batch_import table so it doesn't get amazingly big.
$db = Mage::getSingleton('core/resource')->getConnection('core_write');
$db->query("TRUNCATE TABLE `dataflow_batch_import`");
Mage::log(convert(memory_get_usage()) . " - " . "Table dataflow_batch_import cleaned", null, $logFileName);

//load profile
if ($profileId) {
    $profile->load($profileId);
    if (!$profile->getId()) {
        Mage::getSingleton('adminhtml/session')->addError('ERROR: Could not load profile');
    }
}
Mage::register('current_convert_profile', $profile);

//run the profile
Mage::log(convert(memory_get_usage()) . " - " . "Preparing profile...", null, $logFileName);
$profile->run();
Mage::log(convert(memory_get_usage()) . " - " . "...Done", null, $logFileName);

//get to work
$batchModel = Mage::getSingleton('dataflow/batch');
if ($batchModel->getId()) {
    //echo "getId ok\n";
    if ($batchModel->getAdapter()) {

        //echo "getAdapter ok\n";

        $batchId = $batchModel->getId();
        Mage::log(convert(memory_get_usage()) . " - " . "Loaded batch id $batchId", null, $logFileName);

        $batchImportModel   = $batchModel->getBatchImportModel();
        $importIds          = $batchImportModel->getIdCollection();
        $batchModel         = Mage::getModel('dataflow/batch')->load($batchId);
        $adapter            = Mage::getModel($batchModel->getAdapter());
        $postdata           = array();
        $postnum            = 0;
        $totalproducts      = count($importIds);

        Mage::log(convert(memory_get_usage()) . " - 0/{$totalproducts}", null, $logFileName);
        foreach ($importIds as $importId) {
            //echo "importing $importId\n";
            $recordCount++;
            $postdata[] = "rows[]=$importId";
            //echo "$importId ";
            if ($recordCount % $atOnce == 0 || $recordCount == $totalproducts) {
                $postnum++;
                $postdata[] = "batch_id=$batchId";
                $postdata[] = "form_key=$formKey";
                $postdatastring = implode('&', $postdata);
                $postdata = array();
                //print_r($postdatastring);
                Mage::log(convert(memory_get_usage()) . " - Start cURL request #$postnum", null, $logFileName);
                $ch = curl_init();
                curl_setopt($ch, CURLOPT_URL, $url . "index.php/admin/system_convert_profile/batchRun/?isAjax=true");
                curl_setopt($ch, CURLOPT_TIMEOUT, 200);
                curl_setopt($ch, CURLOPT_COOKIE, "adminhtml=$sessionId");
                curl_setopt($ch, CURLOPT_POST, 1);
                curl_setopt($ch, CURLOPT_POSTFIELDS, $postdatastring);
                curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
                $buffer = curl_exec($ch);
                if (empty($buffer)) {
                    Mage::log(convert(memory_get_usage()) . " - {$recordCount}/{$totalproducts} - Response is empty - ERROR" . curl_error($ch), null, $logFileName);
                } else {
                    $result = json_decode($buffer);
                    Mage::log(convert(memory_get_usage()) . " - {$recordCount}/{$totalproducts} [$buffer]", null, $logFileName);
                    if (@count($result->errors)) {
                        foreach ($result->errors as $error) {
                            Mage::log(convert(memory_get_usage()) . " - ERROR: $error", null, $logFileName);
                        }
                    }
                }
                curl_close($ch);
            }
        }

        foreach ($profile->getExceptions() as $e) {
            Mage::log(convert(memory_get_usage()) . " - " . $e->getMessage(), null, $logFileName);
        }
    }
}

Mage::log(convert(memory_get_usage()) . " - " . "Completed!", null, $logFileName);
?>

mag_login.php:

<?php

/**
 * Path to the root of your magento installation.
 * include traing slash.
 */
$root = '/absolute/path/to/your/magento/root/'

/**
 * Backend username that has the rights to import products.
 */
$username = "username";

/**
 * Password
 */
$password = "password";

 /**
  * DO NOT EDIT BELOW THIS LINE
  */

require_once $root.'app/Mage.php';
umask(0);
Mage::app();

$user = Mage::getModel('admin/user');
if ($user->authenticate($username, $password)) 
{    
    Mage::getSingleton('admin/session')->setUser($user);

    if ($user->getId())
    {
        if (Mage::getSingleton('adminhtml/url')->useSecretKey()) {
            Mage::getSingleton('adminhtml/url')->renewSecretUrls();
        }
        $session = Mage::getSingleton("admin/session");

        //Change the owner of the session file on root/var/session/ to the user that runs the webserver
        //exec("chown nobody:nobody {$root}var/session/sess_".$session->getEncryptedSessionId());

        echo json_encode(array('sessionId' => $session->getEncryptedSessionId(), 'formKey' => Mage::getSingleton('core/session')->getFormKey()));
    }
}
?>

I found these scripts some time ago and altered them for my own needs.

Answer 2

I found this solution : http://www.maximehuran.fr/lancer-plusieurs-dataflow-magento-en-sequence-dans-un-cron/

Put this code after each dataflow

<?php
Mage::getSingleton('dataflow/batch')->delete();
$registryKey = '_singleton/dataflow/batch';
if (Mage::registry($registryKey)) {
    Mage::unregister($registryKey);
}