diff --git a/README.md b/README.md new file mode 100644 index 0000000..35b8328 --- /dev/null +++ b/README.md @@ -0,0 +1,126 @@ +XML Product Import +============== +Converts XML import files into data that is then imported using [AvS\_FastSimpleImport](https://github.com/avstudnitz/AvS_FastSimpleImport) + +Information +----------- +- Version: 0.0.1 +- [Github](https://github.com/code4business/xmlimport) + +Description +----------- +This is a product importer that transforms product data from XML files into data array compatible with [AvS\_FastSimpleImport](https://github.com/avstudnitz/AvS_FastSimpleImport) +and uses it to import that data. The importer processes all XML files from "inbox" directory and moves them to an error or success directory, depending on import result. Any errors +encountered during import are sent by email. + +Any categories that are in the import file but do not yet exist in the shop can be automatically created. +Missing attributes can also be created but they will have to be put into apropriate attribute sets manually. Since there is no way to tell what type the data +should be, all missing attributes are created as select type. + +Features +-------- +- Import any product type +- Create missing categories +- Create missing attributes +- Send Email with missing attributes +- Send Email with errors encountered + +XML format +---------- +This sample XML-file explanis the basic structure: +```xml + + + + + + STORE_CODE + + + + + + + + + value + + value + + + + + value + + value + + + + + + + default scope value + + store scope value + + + + +``` + +This are the meanings of the XML nodes: +- **stores**: Each item inside this node is a store code. Only data for these stores will be read from each attribute for this product. +- **simple_data**: Each element inside here is a simple attribute code. +- **SIMPLE\_ATTRIBUTE\_CODE**: Represents a single simple attribute. The attribute values go to one or more children nodes. +These children nodes need to match the ones defined in `` element or they will not be read. For default scope, +Use the node `` +- **complex_data**: Complex attributes go here. These are attributes that can have multiple values like categories, media images, related products +- **enum**: Collection of attributes that belong together. +- **enum item**: These represent each of the multiple values for a complex attribute. For example, each category or an associated product would be a new item. +- **enum item ATTRIBUTE_CODE**: Each item is collection of attributes that represent a complex attribute. For example, a media image can be defined by up to +5 diffrent attributes. It is important that each element has the same children names and count. + + +Configuration options +--------------------- +Most options should be self-explanatory. + +- Append suffix to duplicates +By default this is set to yes, which is also the Magento behavior. This means that an image with existing name will be saved with a numbered suffix added to it. +If set to no, images will not be renamed. Additionally, image with the same name will only be copied if it is newer. + +- Ignored new attributes +List of attribute codes that will not be treated as missing. + +Compatibility +------------- +- Magento CE >= 1.8.0.0 +- Magento EE >= 1.13.1 + +Current Limitations +------------------- +- When creating missing categories, only one root category is used. In a multi-website shop this would only work for one at a time. + +Contribution +------------ +Any contribution is appreciated, just open a pull request. If possible stick to the following coding rules: + +- Keep your code as simple and as short as possible +- Use speaking method and variable names - this is a very important (or the primary) source of documentation +- Use observers instead of rewrites wherever possible +- Do not duplicate code; so if you copy code from one place to another you are properly doing something wrong +- Only use comments inside methods if the code is really hard to understand and you cannot make it easier; please comment the methods however +- Use sentences for your commit-messages that start with a verb in past tense and end with a dot, e.g. "Add modman file." + + + +Uninstalation +------------- +1. Remove extension files +2. Delete transactional email templates "Product Import Error Report" and "Product Import Missing Attributes Report" + +License +------- +[Open Software Licence 3.0 (OSL-3.0)](http://opensource.org/licenses/osl-3.0.php) diff --git a/examples/example_product_import.xml b/examples/example_product_import.xml new file mode 100644 index 0000000..144c9a0 --- /dev/null +++ b/examples/example_product_import.xml @@ -0,0 +1,619 @@ + + + + + english + french + german + + + + + <_media_image>/images/test_image1.jpg + <_media_is_disabled>0 + + + <_media_image>/images/test_image2.jpg + <_media_is_disabled>0 + + + + + <_category>Importer Category + + + <_category>Importer Category/Subcategory + + + + + <_product_websites>base + + + + + + SIMPLE-1 + + <_attribute_set> + Default + + <_type> + simple + + + Import Simple product + Import English Simple product + Import French Simple product + Import German Einfaches produkt + + + 10 + + + Test description + + + Test short description + + + 1 + + + /images/test_image1.jpg + + + /images/test_image1.jpg + + + test + + + 50 + + + 1 + + + + + + english + french + german + + + + + <_media_image>/images/test_image1.jpg + <_media_is_disabled>0 + + + + + <_category>Importer Category + + + <_category>Importer Category/Subcategory + + + + + <_product_websites>base + + + + + + SIMPLE-2 + + <_attribute_set> + Default + + <_type> + simple + + + Import Simple product 2 + Import English Simple product 2 + Import French Simple product 2 + Import German Einfaches produkt 2 + + + 20 + + + Test description + + + Test short description + + + 1 + + + test + + + 55 + + + 1 + + + + + + english + french + german + + + + + <_media_image>/images/test_image1.jpg + <_media_is_disabled>0 + + + + + <_category>Importer Category + + + <_category>Importer Category/Subcategory + + + + + <_product_websites>base + + + + + + SIMPLE-3 + + <_attribute_set> + Default + + <_type> + simple + + + Import Simple product 3 + Import English Simple product 3 + Import French Simple product 3 + Import German Einfaches produkt 3 + + + 30 + + + Test description + + + Test short description + + + 1 + + + /images/test_image1.jpg + + + /images/test_image1.jpg + + + test + + + 56 + + + 1 + + + + + + english + french + german + + + + + <_media_image>/images/test_image1.jpg + <_media_is_disabled>0 + + + + + <_category>Importer Category + + + + + <_product_websites>base + + + + + <_associated_sku>SIMPLE-1 + <_associated_position>1 + + + <_associated_sku>SIMPLE-2 + <_associated_position>2 + + + + + + GROUPED-1 + + <_attribute_set> + Default + + <_type> + grouped + + + Import Grouped product + Import English Grouped product + Import French Grouped product + Import German Gruppierte produkt + + + 30 + + + Test description + + + Test short description + + + 1 + + + /images/test_image1.jpg + + + /images/test_image1.jpg + + + test + + + 56 + + + 1 + + + + + + english + french + german + + + + + <_media_image>/images/test_image1.jpg + <_media_is_disabled>0 + + + + + <_category>Importer Category/Subcategory + + + + + <_product_websites>base + + + + + + SIMPLE-4 + + <_attribute_set> + default_configurable + + <_type> + simple + + + Import Simple product 4 + Import English Simple product 4 + Import French Simple product 4 + Import German Einfaches produkt 4 + + + 40 + + + Test description + + + Test short description + + + 1 + + + /images/test_image1.jpg + + + /images/test_image1.jpg + + + test + + + 50 + + + 1 + + + green + + + + + + english + french + german + + + + + <_media_image>/images/test_image1.jpg + <_media_is_disabled>0 + + + + + <_category>Importer Category + + + + + <_product_websites>base + + + + + + SIMPLE-5 + + <_attribute_set> + default_configurable + + <_type> + simple + + + Import Simple product 5 + Import English Simple product 5 + Import French Simple product 5 + Import German Einfaches produkt 5 + + + 50 + + + Test description + + + Test short description + + + 1 + + + test + + + 55 + + + 1 + + + red + + + + + + english + french + german + + + + + <_media_image>/images/test_image1.jpg + <_media_is_disabled>0 + + + + + <_category>Importer Category + + + <_category>Importer Category/Subcategory + + + + + <_product_websites>base + + + + + <_super_products_sku>SIMPLE-4 + + + <_super_products_sku>SIMPLE-5 + + + + + <_super_attribute_code>color + + + + + + CONFIGURABLE-1 + + <_attribute_set> + default_configurable + + <_type> + configurable + + + Import configurable product + Import English configurable product + Import French configurable product + Import German Konfigurable produkt + + + 60 + + + Test description + + + Test short description + + + 1 + + + /images/test_image1.jpg + + + /images/test_image1.jpg + + + test + + + 56 + + + 1 + + + + + + english + french + german + + + + + <_media_image>/images/test_image1.jpg + <_media_is_disabled>0 + + + + + <_category>Importer Category + + + + + <_product_websites>base + + + + + <_bundle_option_type>select + <_bundle_option_position>1 + <_bundle_option_required>1 + + + + + <_bundle_option_title>Option one + <_bundle_product_sku>SIMPLE-1 + <_bundle_product_is_default>true + <_bundle_product_qty>5 + <_bundle_product_price_value>10 + <_bundle_product_can_change_qty>0 + + + <_bundle_option_title>Option one + <_bundle_product_sku>SIMPLE-2 + <_bundle_product_is_default>false + <_bundle_product_qty>1 + <_bundle_product_price_value>20 + <_bundle_product_can_change_qty>1 + + + + + + BUNDLE-1 + + <_attribute_set> + Default + + <_type> + bundle + + + Import bundle product + Import English bundle product + Import French bundle product + Import German bundle produkt + + + 70 + + + 1 + + + as low as + + + Test description + + + Test short description + + + 1 + + + /images/test_image1.jpg + + + /images/test_image1.jpg + + + test + + + 59 + + + 1 + + + + \ No newline at end of file diff --git a/modman b/modman new file mode 100644 index 0000000..ea2916a --- /dev/null +++ b/modman @@ -0,0 +1,3 @@ +src/app/code/local/C4B/XmlImport/ app/code/local/C4B/XmlImport/ +src/app/design/frontend/base/default/template/c4b/xmlimport/ app/design/frontend/base/default/template/c4b/xmlimport/ +src/app/etc/modules/C4B_XmlImport.xml app/etc/modules/C4B_XmlImport.xml \ No newline at end of file diff --git a/src/app/code/community/C4B/XmlImport/Block/Adminhtml/System/Config/Email.php b/src/app/code/community/C4B/XmlImport/Block/Adminhtml/System/Config/Email.php new file mode 100644 index 0000000..d574547 --- /dev/null +++ b/src/app/code/community/C4B/XmlImport/Block/Adminhtml/System/Config/Email.php @@ -0,0 +1,33 @@ + + * @copyright code4business Software GmbH + */ +class C4B_XmlImport_Block_Adminhtml_System_Config_Email extends Mage_Adminhtml_Block_System_Config_Form_Field_Array_Abstract +{ + /** + * Add columns to the form for Invoice Texts. + * */ + public function __construct() + { + $this->addColumn('name', array( + 'label' => Mage::helper('xmlimport')->__('Name'), + 'style' => 'width:100px', + )); + + $this->addColumn('email', array( + 'label' => Mage::helper('xmlimport')->__('Email'), + 'style' => 'width:200px', + )); + + $this->_addAfter = false; + $this->_addButtonLabel = Mage::helper('xmlimport')->__('Add'); + parent::__construct(); + } +} diff --git a/src/app/code/community/C4B/XmlImport/Block/Importerror.php b/src/app/code/community/C4B/XmlImport/Block/Importerror.php new file mode 100644 index 0000000..1eeb99c --- /dev/null +++ b/src/app/code/community/C4B/XmlImport/Block/Importerror.php @@ -0,0 +1,81 @@ + + * @copyright code4business Software GmbH + */ +class C4B_XmlImport_Block_Importerror extends Mage_Core_Block_Template +{ + /** + * Constructor, sets the template. + * @see Mage_Core_Block_Template::_construct() + */ + public function _construct() + { + $this->setTemplate('c4b/xmlimport/importreport.phtml'); + } + + /** + * Gets the original imported file count in the result import. + * @return int + */ + public function getFileCount() + { + $resultData = $this->getData('result_data'); + return $resultData['count_import_files']; + } + + /** + * Gets the number of imported files that have errors. + * @return int + */ + public function getErrorFileCount() + { + $resultData = $this->getData('result_data'); + return $resultData['count_error_import_files']; + } + + /** + * Returns all the file import reports that have errors. + * @return array C4B_CommandLineImporter_Model_Results + */ + public function getErrors() + { + $resultData = $this->getData('result_data'); + return $resultData['errors']; + } + + /** + * Returns the timestamp when importing was started. + * @return string + */ + public function getStartTime() + { + $resultData = $this->getData('result_data'); + return $resultData['start_time']; + } + + /** + * Returns the time taken to import the files in seconds. + * @return int + */ + public function getTimeTaken() + { + $resultData = $this->getData('result_data'); + return $resultData['time_taken']; + } + + /** + * Returns the total amount of memory used in importing + * @return string + */ + public function getMemoryUsed() + { + $resultData = $this->getData('result_data'); + return $resultData['memory_used']; + } +} diff --git a/src/app/code/community/C4B/XmlImport/Block/Missingattributes.php b/src/app/code/community/C4B/XmlImport/Block/Missingattributes.php new file mode 100644 index 0000000..935dd5a --- /dev/null +++ b/src/app/code/community/C4B/XmlImport/Block/Missingattributes.php @@ -0,0 +1,39 @@ + + * @copyright code4business Software GmbH + */ +class C4B_XmlImport_Block_Missingattributes extends Mage_Core_Block_Template +{ + /** + * Constructor, sets the template. + * @see Mage_Core_Block_Template::_construct() + */ + public function _construct() + { + $this->setTemplate('c4b/xmlimport/missing_attributes.phtml'); + } + + /** + * Get the array of created attributes + * @return array + */ + public function getAttributesList() + { + return $this->getData('attributes'); + } + + /** + * If the notice about assigning created attributes to attribute sets should be displayed + * @return boolean + */ + public function getAreAttributesCreated() + { + return $this->getData('attributes_created'); + } +} diff --git a/src/app/code/community/C4B/XmlImport/Helper/Data.php b/src/app/code/community/C4B/XmlImport/Helper/Data.php new file mode 100644 index 0000000..badd4a9 --- /dev/null +++ b/src/app/code/community/C4B/XmlImport/Helper/Data.php @@ -0,0 +1,101 @@ + + * @copyright code4business Software GmbH + */ +class C4B_XmlImport_Helper_Data extends Mage_Core_Helper_Abstract +{ + protected $_utf8ConversionMap = array( 0x0, 0x2FFFF, 0, 0xFFFF ); + /** + * Return the given nodes first level children as an array. Excludes attributes. + * Format: 'node_name' => 'node_value' + * @param SimpleXMLElement $parentNode + * @return array + */ + public function xmlChildrenToArray($parentNode) + { + $arrayToReturn = array(); + foreach($parentNode as $nodeName => $childNode) + { + $arrayToReturn[$nodeName] = $childNode->__toString(); + } + return $arrayToReturn; + } + + /** + * Decode the numeric entities in the given string and return it. + * @param string $encodedString + * @return string + */ + public function parseNumericEntities($encodedString) + { + return mb_decode_numericentity($encodedString, $this->_utf8ConversionMap, 'UTF-8'); + } + + /** + * Convert size to human readable + * + * @param int $number + * @return string + */ + public function humanSize($number) + { + if ($number < 1000) { + return sprintf('%d b', $number); + } else if ($number >= 1000 && $number < 1000000) { + return sprintf('%.2fKb', $number / 1000); + } else if ($number >= 1000000 && $number < 1000000000) { + return sprintf('%.2fMb', $number / 1000000); + } else { + return sprintf('%.2fGb', $number / 1000000000); + } + } + + /** + * Gets the selected directory path and creates it if it doesn't exist. + * @param string $type + * @return string + */ + public function getDirectory($type = 'base') + { + /* @var $dirCreator Mage_Core_Model_Config_Options */ + $dirCreator = Mage::getModel('core/config_options'); + + $path = Mage::getBaseDir(); + switch($type) + { + case 'success': + $path .= DS . Mage::getStoreConfig(C4B_XmlImport_Model_Importer::XML_PATH_IMPORT_SUCCESS_DIRECTORY); + break; + case 'error': + $path .= DS . Mage::getStoreConfig(C4B_XmlImport_Model_Importer::XML_PATH_IMPORT_ERROR_DIRECTORY); + break; + case 'base': + $path .= DS . Mage::getStoreConfig(C4B_XmlImport_Model_Importer::XML_PATH_IMPORT_DIRECTORY); + default: + break; + } + $dirCreator->createDirIfNotExists($path); + return $path; + } + + /** + * Format the errors to a single array + * @param array $errors + */ + public function formatErrors($errors) + { + $messages = array(); + $n = 0; + foreach ($errors as $message => $rows) + { + $messages[]= sprintf("\t%d: %s (rows: %s)", ++$n, $message, join(',', $rows)); + } + return join("\n", $messages); + } +} \ No newline at end of file diff --git a/src/app/code/community/C4B/XmlImport/Model/AttributeCreator.php b/src/app/code/community/C4B/XmlImport/Model/AttributeCreator.php new file mode 100644 index 0000000..d472312 --- /dev/null +++ b/src/app/code/community/C4B/XmlImport/Model/AttributeCreator.php @@ -0,0 +1,138 @@ + + * @copyright code4business Software GmbH + */ +class C4B_XmlImport_Model_AttributeCreator +{ + protected $_existingAttributes = array(); + protected $_missingAttributes = array(); + + const XML_PATH_PREPROCESSING_CREATE_ATTRIBUTES = 'c4b_xmlimport/preprocessing/create_attributes'; + const XML_PATH_PREPROCESSING_IGNORED_NEW_ATTRIBUTES = 'c4b_xmlimport/preprocessing/ignored_new_attributes'; + + const EVENT_MISSING_ATTRIBUTE_CREATED = 'c4b_xmlimport_missing_attribute_created'; + + /** + * Default constructor. + */ + public function __construct() + { + //These attributes have a particular meaning, and don't directly map to one attribute + $particularAttributes = array( + '_store', '_attribute_set', '_type', '_category', '_root_category', '_product_websites', + '_tier_price_website', '_tier_price_customer_group', '_tier_price_qty', '_tier_price_price', + '_links_related_sku', '_group_price_website', '_group_price_customer_group', '_group_price_price', + '_links_related_position', '_links_crosssell_sku', '_links_crosssell_position', '_links_upsell_sku', + '_links_upsell_position', '_custom_option_store', '_custom_option_type', '_custom_option_title', + '_custom_option_is_required', '_custom_option_price', '_custom_option_sku', '_custom_option_max_characters', + '_custom_option_sort_order', '_custom_option_file_extension', '_custom_option_image_size_x', + '_custom_option_image_size_y', '_custom_option_row_title', '_custom_option_row_price', + '_custom_option_row_sku', '_custom_option_row_sort', '_media_attribute_id', '_media_image', '_media_lable', + '_media_position', '_media_is_disabled', + //Stock Data attributes + 'manage_stock', 'use_config_manage_stock', 'qty', 'min_qty', 'use_config_min_qty', 'min_sale_qty', 'max_sale_qty', + 'use_config_max_sale_qty', 'is_qty_decimal', 'backorders', 'use_config_backorders', 'notify_stock_qty', + 'use_config_notify_stock_qty', 'enable_qty_increments', 'use_config_enable_qty_inc', 'qty_increments', + 'use_config_qty_increments', 'is_in_stock', 'low_stock_date', 'stock_status_changed_auto', 'is_decimal_divided' + ); + + $attributesToIgnore = explode(',', Mage::getStoreConfig(self::XML_PATH_PREPROCESSING_IGNORED_NEW_ATTRIBUTES)); + + $existingAttributes = Mage::getSingleton('eav/config')->getEntityType(Mage_Catalog_Model_Product::ENTITY)->getAttributeCollection(); + + /* @var $existingAttributes Mage_Catalog_Model_Resource_Eav_Mysql4_Attribute_Collection */ + foreach ( $existingAttributes as $attribute ) + { + $this->_existingAttributes[strtolower($attribute->getAttributeCode())] = 1; + } + foreach ( $particularAttributes as $attributeCode ) + { + $this->_existingAttributes[$attributeCode] = 1; + } + foreach ( $attributesToIgnore as $attributeCode ) + { + $this->_existingAttributes[strtolower($attributeCode)] = 1; + } + + + } + + /** + * Creates the specified attribute if it doesn't exist. + * @param string $attributeName + * @return C4B_XmlImport_Model_AttributeCreator | int + */ + public function createIfNotExists($attributeCode) + { + $attributeCode = strtolower($attributeCode); + $messageHandler = Mage::getSingleton('xmlimport/messageHandler'); + + if( isset($this->_existingAttributes[$attributeCode]) ) + { + return true; + } + + $createAttributes = ( + Mage::getStoreConfig(self::XML_PATH_PREPROCESSING_CREATE_ATTRIBUTES) == + C4B_XmlImport_Model_Source_Attribute_ProcessingMode::PROCESSING_MODE_CREATE_AND_INFORM + ); + + if(!$createAttributes) + { + if(!array_key_exists($attributeCode, $this->_missingAttributes)) + { + $messageHandler->addNotice("Attribute '{$attributeCode}' does not exist and won't be imported."); + } + $this->_missingAttributes[$attributeCode] = 1; + return false; + } + + + $entityTypeId = Mage::getModel('catalog/product')->getResource()->getEntityType()->getId(); + $newAttribute = Mage::getModel('eav/entity_attribute'); + $newAttribute->setData(array( + 'entity_type_id' => $entityTypeId, + 'attribute_code' => $attributeCode, + 'frontend_label' => $attributeCode, + 'frontend_input' => 'select', + 'backend_type' => $newAttribute->getBackendTypeByInput('select'), + 'source' => 'eav/entity_attribute_source_table', + 'is_global' => '1', + 'is_visible' => '1', + 'is_visible_on_front' => '1', + 'is_user_defined' => '1', + 'is_searchable' => '0', + 'is_filterable' => '0', + 'is_filterable_in_search' => '0', + 'is_comparable' => '1', + 'is_configurable' => '1', + 'apply_to' => 'simple', + 'default' => '0' + )); + + Mage::dispatchEvent( self::EVENT_MISSING_ATTRIBUTE_CREATED, array('attribute' => $newAttribute) ); + + $newAttribute->save(); + $this->_missingAttributes[$attributeCode] = 1; + $this->_existingAttributes[$attributeCode] = 1; + + $messageHandler->addNotice("Created attribute {$attributeCode}."); + + return true; + } + + /** + * Get the missing attribute codes. + * @return array + */ + public function getMissingAttributes() + { + return array_keys( $this->_missingAttributes ); + } +} diff --git a/src/app/code/community/C4B/XmlImport/Model/CategoryCreator.php b/src/app/code/community/C4B/XmlImport/Model/CategoryCreator.php new file mode 100644 index 0000000..bb775f8 --- /dev/null +++ b/src/app/code/community/C4B/XmlImport/Model/CategoryCreator.php @@ -0,0 +1,173 @@ + + * @copyright code4business Software GmbH + */ +class C4B_XmlImport_Model_CategoryCreator +{ + const EVENT_MISSING_CATEGORY_CREATED = 'c4b_xmlimport_missing_category_created'; + + const XML_PATH_ROOT_CATEGORY_ID = 'c4b_xmlimport/preprocessing/root_category'; + + protected $_categories = array(); + protected $_errors = array(); + + /** + * Default constructor. + */ + public function __construct() + { + $this->_initCategories(); + } + + /** + * Creates the specified category if it doesn't exist. + * @param string $categoryName + * @return C4B_XmlImport_Model_CategoryCreator|int + */ + public function createIfItNotExists($categoryName) + { + if( !isset($this->_categories[$categoryName]) ) + { + return $this->_createCategoryRecursively($categoryName) != null; + } + return true; + } + + /** + * Retreive an array of messages that were generated during creation of categories and delete. + * @return array + */ + public function getErrors() + { + $errors = $this->_errors; + $this->_errors = array(); + return $errors; + } + + /** + * Initialize an array with all category paths. Code taken from Avs_FastSimpleImport + * @see AvS_FastSimpleImport_Model_Import_Entity_Product::_initCategories() + */ + protected function _initCategories() + { + $collection = Mage::getResourceModel('catalog/category_collection')->addNameToResult(); + /* @var $collection Mage_Catalog_Model_Resource_Eav_Mysql4_Category_Collection */ + foreach ($collection as $category) { + $structure = explode('/', $category->getPath()); + $pathSize = count($structure); + if ($pathSize > 2) { + $path = array(); + $this->_categories[implode('/', $path)] = $category->getId(); + for ($i = 1; $i < $pathSize; $i++) { + $item = $collection->getItemById($structure[$i]); + if ($item instanceof Varien_Object) { + $path[] = $item->getName(); + } + } + + // additional options for category referencing: name starting from base category, or category id + $this->_categories[implode('/', $path)] = $category->getId(); + array_shift($path); + $this->_categories[implode('/', $path)] = $category->getId(); + $this->_categories[$category->getId()] = $category->getId(); + } + } + } + + /** + * Create nonexistent category and all nonexistent parents recursively. + * @param string $categoryName Name of category E.g. Kategorien/Keramic/Hersteller5 + * @return int|null + */ + protected function _createCategoryRecursively($categoryName) + { + /* @var $messageHandler C4B_XmlImport_Model_MessageHandler */ + $messageHandler = Mage::getSingleton('xmlimport/messageHandler'); + if (trim($categoryName) == '') + { + return null; + } + + $categoryPathArray = explode('/', $categoryName); + + // Check that category path does not have empty element names. + foreach($categoryPathArray as $categoryNameItem) + { + if(empty($categoryNameItem)) + { + $this->_errors[] ="Category [{$categoryName}] can not have empty path parts."; + return null; + } + } + $newCategoryName = array_pop($categoryPathArray); + $categoryParentPath = implode('/',$categoryPathArray); + + // If current category's parent category does not exist - create it recursively. + if( !isset($this->_categories[$categoryParentPath]) ) + { + if( $this->_createCategoryRecursively($categoryParentPath) == null ) + { + return null; + } + } + + $pathPrefix = ''; + // Recreate category path. + $categoryParentIds = array(); + foreach($categoryPathArray as $categoryPathItem) //Put check for existence of category id + { + if(isset($this->_categories[$pathPrefix . $categoryPathItem])) + { + $categoryParentIds[] = $this->_categories[$pathPrefix . $categoryPathItem]; + }else{ + $this->_errors[] = "Category path does not have needed links {$pathPrefix} {$categoryPathItem}"; + } + $pathPrefix .= $categoryPathItem . '/'; + } + + if( count($categoryParentIds) > 0 ) + { + $categoryParentPath = $this->_getCategoryPathPrefix() . '/' . implode('/', $categoryParentIds); + }else + { + $categoryParentPath = $this->_getCategoryPathPrefix(); + } + // Create a category + /* @var $category Mage_Catalog_Model_Category */ + $category = Mage::getModel('catalog/category'); + $category->setName($newCategoryName); + $category->setIsActive(true); + $category->setPath($categoryParentPath); + $category->setDisplayMode( Mage_Catalog_Model_Category::DM_PRODUCT ); + + Mage::dispatchEvent( self::EVENT_MISSING_CATEGORY_CREATED, array('category' => $category) ); + + try { + $category->save(); + $messageHandler->addNotice("Created category {$categoryName}"); + } catch (Exception $e) + { + Mage::logException($e); + Mage::throwException("Category with name {$newCategoryName} and path {$categoryParentPath} can not be saved."); + } + + $categoryId = $category->getId(); + $this->_categories[$categoryName] = $categoryId; + return $categoryId; + } + + /** + * Get the category path prefix. It consists of the constant 1 and the ID of the root category that is configured. + * @return string + */ + protected function _getCategoryPathPrefix() + { + return '1/' . Mage::getStoreConfig(self::XML_PATH_ROOT_CATEGORY_ID); + } +} \ No newline at end of file diff --git a/src/app/code/community/C4B/XmlImport/Model/Import/Entity/Product.php b/src/app/code/community/C4B/XmlImport/Model/Import/Entity/Product.php new file mode 100644 index 0000000..7ad3d82 --- /dev/null +++ b/src/app/code/community/C4B/XmlImport/Model/Import/Entity/Product.php @@ -0,0 +1,43 @@ + + * @copyright code4business Software GmbH + */ +class C4B_XmlImport_Model_Import_Entity_Product extends AvS_FastSimpleImport_Model_Import_Entity_Product +{ + /** + * Use factory to get fileUplaoder object. + * @see AvS_FastSimpleImport_Model_Import_Entity_Product::_getUploader() + */ + protected function _getUploader() + { + if (is_null($this->_fileUploader)) + { + $this->_fileUploader = Mage::getModel('importexport/import_uploader', null); // Use factory method instead of new + $this->_fileUploader->init(); + + $tmpDir = Mage::getConfig()->getOptions()->getMediaDir() . '/import'; + $destDir = Mage::getConfig()->getOptions()->getMediaDir() . '/catalog/product'; + if (!is_writable($destDir)) { + @mkdir($destDir, 0777, true); + } + // diglin - add auto creation in case folder doesn't exist + if (!file_exists($tmpDir)) { + @mkdir($tmpDir, 0777, true); + } + if (!$this->_fileUploader->setTmpDir($tmpDir)) { + Mage::throwException("File directory '{$tmpDir}' is not readable."); + } + if (!$this->_fileUploader->setDestDir($destDir)) { + Mage::throwException("File directory '{$destDir}' is not writable."); + } + } + return $this->_fileUploader; + } +} diff --git a/src/app/code/community/C4B/XmlImport/Model/Importer.php b/src/app/code/community/C4B/XmlImport/Model/Importer.php new file mode 100644 index 0000000..3a8c75c --- /dev/null +++ b/src/app/code/community/C4B/XmlImport/Model/Importer.php @@ -0,0 +1,223 @@ + + * @copyright code4business Software GmbH + */ +class C4B_XmlImport_Model_Importer +{ + const IMPORT_FILE_EXTENSION = 'xml'; + const XML_PATH_IMPORT_DIRECTORY = 'c4b_xmlimport/general/import_xml_dir'; + const XML_PATH_IMPORT_SUCCESS_DIRECTORY = 'c4b_xmlimport/general/success_dir'; + const XML_PATH_IMPORT_ERROR_DIRECTORY = 'c4b_xmlimport/general/error_dir'; + const XML_PATH_UPLOAD_IMAGE = 'c4b_xmlimport/general/upload_image'; + + const XML_PATH_PREPROCESSING_CREATE_CATEGORIES = 'c4b_xmlimport/preprocessing/create_category'; + + const EVENT_AFTER_DATA_IMPORT = 'c4b_xmlimport_after_data_import'; + + /** + * Collected files for importing + * @var array(string) + */ + protected $_importFiles = array(); + + /** + * Name used when obtaining a lock + * @var string + */ + protected $_lockName = 'c4b_xmlimport_lock'; + + /** + * Run the import process. Returns true on success or false on any error. + * @param boolean $performImport + * @return boolean + */ + public function run() + { + /* @var $messageHandler C4B_XmlImport_Model_MessageHandler */ + $messageHandler = Mage::getSingleton('xmlimport/messageHandler'); + $messageHandler->startReport(); + + $importFiles = $this->_collectImportFiles(); + $success = false; + + if( count($importFiles) == 0 ) + { + $messageHandler->addNotice('No xml files found in import directory.'); + $success = true; + } + else if( count($importFiles) != 0 && Mage::getResourceHelper('xmlimport')->setNamedLock($this->_lockName) ) + { + $this->_processImportFiles($importFiles); + $success = true; + } + else + { + $messageHandler->addError('Could not obtain lock.'); + $messageHandler->addError('Importing will not be performed.'); + } + + $messageHandler->finalizeResults(); + Mage::getResourceHelper('xmlimport')->releaseNamedLock($this->_lockName); + return $success; + } + + /** + * Set the name that will be used to set a DB lock. + * @param string $lockName + * @return C4B_XmlImport_Model_Importer + */ + public function setLockName($lockName) + { + $this->_lockName = $lockName; + return $this; + } + + /** + * Import given XML file. + * @param string $filePath + * @return boolean|C4B_XmlImport_Model_Importer|number + */ + protected function _importFile($filePath) + { + $messageHandler = Mage::getSingleton('xmlimport/messageHandler'); + /* @var $products C4B_XmlImport_Model_Products */ + $products = Mage::getModel('xmlimport/products'); + + $messageHandler->addNotice("Validating file structure."); + $validationResult = $products->validateFile($filePath); + + $success = false; + switch($validationResult) + { + case C4B_XmlImport_Model_Products::VALIDATION_RESULT_FILE_ERROR: + $messageHandler->addError('File has syntax errors.'); + break; + case C4B_XmlImport_Model_Products::VALIDATION_RESULT_NO_ROOT_NODE: + $messageHandler->addError('File is missing the root node.'); + $success = true; + break; + case C4B_XmlImport_Model_Products::VALIDATION_RESULT_NO_PRODUCT_NODES: + $messageHandler->addNotice('File has no product nodes.'); + $success = true; + break; + case C4B_XmlImport_Model_Products::VALIDATION_RESULT_OK: + $messageHandler->addNotice("File structure valid."); + + $messageHandler->addNotice("Preparing data."); + $importData = $products->processFile($filePath); + $messageHandler->addNotice("Data ready."); + + if( !empty($importData) && count($importData) > 0 ) + { + $messageHandler->addNotice('Importing started.'); + if( $this->_importData($importData) ) + { + $messageHandler->addNotice('Importing completed.'); + $success = true; + } else + { + $messageHandler->addErrorsForFile( basename($filePath), ' ', 'Data was not valid for import' ); + $success = false; + } + } + else + { + $messageHandler->addErrorsForFile( basename($filePath), ' ', 'File has no valid product nodes.' ); + $success = false; + } + break; + } + return $success; + } + + /** + * Trigger the import for currently set data. + * @param array $data + */ + protected function _importData($data) + { + $messageHandler = Mage::getSingleton('xmlimport/messageHandler'); + /* @var $fastSimpleImport AvS_FastSimpleImport_Model_Import */ + $fastSimpleImport = Mage::getModel('fastsimpleimport/import'); + $importResult = false; + if (!empty($data)) + { + try + { + $fastSimpleImport->processProductImport($data); + $importResult = ( ($fastSimpleImport->getProcessedRowsCount() > 0) && ($fastSimpleImport->getProcessedRowsCount() > $fastSimpleImport->getInvalidRowsCount()) ); + Mage::dispatchEvent(self::EVENT_AFTER_DATA_IMPORT, array('imported_product_skus' => array_keys($fastSimpleImport->getEntityAdapter()->getNewSku()) )); + }catch(Exception $e) + { + Mage::logException($e); + $messageHandler->addError($e->getMessage()); + } + + if($fastSimpleImport->getErrorsCount() > 0) + { + $messageHandler->addError(Mage::helper('xmlimport')->formatErrors($fastSimpleImport->getErrorMessages())); + } + } + return $importResult; + } + + /** + * Process the collected files by validating and importing them + * @param array + */ + protected function _processImportFiles($importFiles) + { + /* $messageHandler C4B_XmlImport_Model_MessageHandler */ + $messageHandler = Mage::getSingleton('xmlimport/messageHandler'); + /* @var $helper C4B_XmlImport_Helper_Data */ + $helper = Mage::helper('xmlimport'); + foreach ($importFiles as $fileName => $filePath) + { + $messageHandler->addNotice("Processing file {$fileName}."); + if($this->_importFile( realpath($filePath) )) + { + rename( $filePath, $helper->getDirectory('success') . DS . $fileName ); + } else + { + rename( $filePath, $helper->getDirectory('error') . DS . $fileName ); + } + $messageHandler->addNotice("File {$fileName} processed."); + } + return $this; + } + + /** + * Collect import files in the import directory. Returns an array of import files: + * [FILENAME] => FILEPATH + * @return array + */ + protected function _collectImportFiles() + { + $importDir = Mage::helper('xmlimport')->getDirectory(); + /* $messageHandler C4B_XmlImport_Model_MessageHandler */ + $messageHandler = Mage::getSingleton('xmlimport/messageHandler'); + + $files = scandir($importDir); + $importFiles = array(); + foreach($files as $fileName) + { + $filePath = $importDir . DS . $fileName; + $fileParts = pathinfo($filePath); + if($fileName == '.' || $fileName == '..' || is_dir($filePath) || $fileParts['extension'] != C4B_XmlImport_Model_Importer::IMPORT_FILE_EXTENSION) + { + continue; + } + $importFiles[$fileName] = $filePath; + } + $messageHandler->addNotice('Number of files found in the import directory: ' . count($importFiles)); + $messageHandler->setNumberOfImportFiles( count($importFiles) ); + return $importFiles; + } +} diff --git a/src/app/code/community/C4B/XmlImport/Model/MessageHandler.php b/src/app/code/community/C4B/XmlImport/Model/MessageHandler.php new file mode 100644 index 0000000..125c758 --- /dev/null +++ b/src/app/code/community/C4B/XmlImport/Model/MessageHandler.php @@ -0,0 +1,255 @@ + + * @copyright code4business Software GmbH + * + * @method int getStartMicroTime() + * @method C4B_XmlImport_Model_MessageHandler setStartMicroTime(int) + * @method int getStartMem() + * @method C4B_XmlImport_Model_MessageHandler setStartMem(int) + * @method string getStartTimestamp() + * @method C4B_XmlImport_Model_MessageHandler setStartTimestamp(string) + * @method string getTimeTaken() + * @method C4B_XmlImport_Model_MessageHandler setTimeTaken(string) + * @method int getMemoryUsed() + * @method C4B_XmlImport_Model_MessageHandler setMemoryUsed(int) + * @method int getNumberOfImportFiles() + * @method C4B_XmlImport_Model_MessageHandler setNumberOfImportFiles(int) + */ +class C4B_XmlImport_Model_MessageHandler extends Varien_Object +{ + const DEFAULT_LOG_FILE_NAME = 'import.log'; + + const XML_PATH_NOTIFICATIONS_MISSING_ATTRIBUTES_RECIPIENTS = 'c4b_xmlimport/notifications/missing_attributes_recipients'; + const XML_PATH_NOTIFICATIONS_MISSING_ATTRIBUTES_EMAIL_TEMPLATE = 'c4b_xmlimport/notifications/missing_attributes_template'; + const XML_PATH_NOTIFICATIONS_IMPORT_ERROR_RECIPIENTS = 'c4b_xmlimport/notifications/import_error_receipients'; + const XML_PATH_NOTIFICATIONS_IMPORT_ERROR_EMAIL_TEMPLATE = 'c4b_xmlimport/notifications/import_error_template'; + + /** + * + * @var array + */ + protected $_errors = array(); + + /** + * XML error types + * @var array + */ + protected $_xmlErrorMessage = array('none','warning','error','fatal'); + + /** + * Log message as notice. + * @param string $message + * @param boolean $print + */ + public function addNotice($message) + { + return $this->_addMessage($message, Zend_Log::NOTICE); + } + + /** + * Log message as error. + * @param string $message + * @param string $print + */ + public function addError($message) + { + if(is_array($message)) + { + foreach ($message as $singleMessage) + { + $this->addError($singleMessage); + } + } else + { + $this->_addMessage($message, Zend_Log::ERR); + } + } + + /** + * Format libXmlError messages as an array of strings. + * @param array $libXmlError + */ + public function formatXmlParseMessage($libXmlError) + { + $messages = array(); + /* @var $libXmlError $singleError */ + foreach($libXmlError as $singleError) + { + $messages[] = "XML read {$this->_xmlErrorMessage[$singleError->level]}, code [{$singleError->code}], line [{$singleError->line}], column [{$singleError->column}]:\n $singleError->message"; + } + return $messages; + } + + /** + * Save, print and output error messages. Messages are saved per file and product. + * @param string $filename + * @param string $productNumber + * @param array|string $error + */ + public function addErrorsForFile($filename, $productNumber, $error) + { + if(is_array($error)) + { + foreach($error as $singleError) + { + $this->addErrorsForFile($filename, $productNumber, $singleError); + } + } else + { + if( !array_key_exists($filename, $this->_errors) ) + { + $this->_errors[$filename] = array(); + } + if( array_key_exists($productNumber, $this->_errors[$filename]) ) + { + $this->_errors[$filename][$productNumber][] = $error; + } else + { + $this->_errors[$filename][$productNumber] = array($error); + } + $this->addError($error); + } + } + + /** + * Log a message to the logfile and output it to stdout. + * @param string $message + * @param int $level + * @return C4B_XmlImport_Model_MessageHandler + */ + protected function _addMessage($message, $level) + { + $message = '['.strftime('%Y-%m-%d %H:%M:%S').'] ' . $message; + echo $message . "\n"; + Mage::log($message, $level, self::DEFAULT_LOG_FILE_NAME, true); + return $this; + } + + /** + * Get start time for use in email template. + * @return string + */ + public function getStartTime() + { + return $this->getData('start_timestamp'); + } + + /** + * Things to do after import process is done. + * @return C4B_XmlImport_Model_MessageHandler + */ + public function finalizeResults() + { + $this->setTimeTaken(microtime(true) - $this->getStartMicroTime()); + $this->setMemoryUsed(Mage::helper('xmlimport')->humanSize(memory_get_usage() - $this->getStartMem())); + + $this->addNotice('Import process done'); + $this->addNotice("Time taken: {$this->getTimeTaken()} s"); + $this->addNotice("Memory used: {$this->getMemoryUsed()}"); + $this->_sendErrorReport(); + $this->_sendAttributesReport(); + return $this; + } + + /** + * Log start of import process + * @return C4B_XmlImport_Model_Importer_Result + */ + public function startReport() + { + $this->setStartMicroTime( microtime(true) ); + $this->setStartTimestamp( strftime('%Y-%m-%d %H:%M:%S'),$this->getStartMicroTime() ); + $this->setStartMem(memory_get_usage()); + $this->addNotice('Import started.'); + $this->addNotice('System process ID: ' . getmypid()); + return $this; + } + + /** + * Reterive and format the recipients from store config or false if there aren't any. + * @param string $path + * @return array|boolean + */ + protected function _getRecipients($path) + { + $recipients = unserialize(Mage::getStoreConfig($path)); + if($recipients !== null && is_array($recipients) && count($recipients) > 0) + { + $transformed = array(); + foreach($recipients as $recipient) + { + $transformed['names'][] = $recipient['name']; + $transformed['emails'][] = $recipient['email']; + } + return $transformed; + } + return false; + } + + /** + * Send an email with error report for the product import to recipients who are configured. + * @return C4B_XmlImport_Model_MessageHandler + */ + protected function _sendErrorReport() + { + $recipients = $this->_getRecipients(self::XML_PATH_NOTIFICATIONS_IMPORT_ERROR_RECIPIENTS); + if( count($this->_errors) == 0 || !$recipients) + { + return $this; + } + $results = array( + 'start_time' => $this->getStartTimestamp(), + 'time_taken' => $this->getTimeTaken(), + 'memory_used' => $this->getMemoryUsed(), + 'count_import_files' => $this->getNumberOfImportFiles(), + 'count_error_import_files' => count($this->_errors), + 'errors' => $this->_errors + ); + /* @var $email Mage_Core_Model_Email_Template */ + $email = Mage::getModel('core/email_template'); + $email->sendTransactional( + Mage::getStoreConfig(self::XML_PATH_NOTIFICATIONS_IMPORT_ERROR_EMAIL_TEMPLATE), + 'general', + $recipients['emails'], + $recipients['names'], + array('results' => $results, 'startTime' => $this->getStartTimestamp()) + ); + } + + /** + * Send an email with missing attributes report to recipients who are configured. + */ + protected function _sendAttributesReport() + { + /* @var $attributeCreator C4B_XmlImport_Model_AttributeCreator */ + $attributeCreator = Mage::getSingleton('xmlimport/attributeCreator'); + $missingAttributes = $attributeCreator->getMissingAttributes(); + + $recipients = $this->_getRecipients(self::XML_PATH_NOTIFICATIONS_MISSING_ATTRIBUTES_RECIPIENTS); + + if( count($missingAttributes) == 0 || !$recipients) + { + return $this; + } + + /* @var $email Mage_Core_Model_Email_Template */ + $email = Mage::getModel('core/email_template'); + $email->sendTransactional( + Mage::getStoreConfig(self::XML_PATH_NOTIFICATIONS_MISSING_ATTRIBUTES_EMAIL_TEMPLATE), + 'general', + $recipients['emails'], + $recipients['names'], + array('attributes' => $missingAttributes, + 'attributes_created' => Mage::getStoreConfig(C4B_XmlImport_Model_AttributeCreator::XML_PATH_PREPROCESSING_CREATE_ATTRIBUTES), + 'startTime' => $this->getStartTimestamp() + ) + ); + } +} diff --git a/src/app/code/community/C4B/XmlImport/Model/Products.php b/src/app/code/community/C4B/XmlImport/Model/Products.php new file mode 100644 index 0000000..6670fa8 --- /dev/null +++ b/src/app/code/community/C4B/XmlImport/Model/Products.php @@ -0,0 +1,91 @@ + + * @copyright code4business Software GmbH + */ +class C4B_XmlImport_Model_Products +{ + const VALIDATION_RESULT_FILE_ERROR = 0; + const VALIDATION_RESULT_OK = 1; + const VALIDATION_RESULT_NO_ROOT_NODE = 2; + const VALIDATION_RESULT_NO_PRODUCT_NODES = 3; + + const XML_NODE_NAME_ROOT = 'products'; + const XML_NODE_NAME_PRODUCT = 'product'; + + /** + * Validate XML file for syntax errors and basic node presence. + * @param string $filePath + * @return string + */ + public function validateFile($filePath) + { + $messageHandler = Mage::getSingleton('xmlimport/messageHandler'); + try + { + libxml_use_internal_errors(true); + $xml = simplexml_load_file($filePath); + $messageHandler->addErrorsForFile( basename($filePath), ' ', $messageHandler->formatXmlParseMessage(libxml_get_errors()) ); + } catch(Exception $e) + { + return self::VALIDATION_RESULT_FILE_ERROR; + } + if(!$xml) + { + return self::VALIDATION_RESULT_NO_ROOT_NODE; + } + if(!property_exists($xml, self::XML_NODE_NAME_PRODUCT)) + { + return self::VALIDATION_RESULT_NO_PRODUCT_NODES; + } + + return self::VALIDATION_RESULT_OK; + } + + /** + * Processes given xml file by iterating over product nodes and extracting data into array + * @param string $filePath + * @return boolean + */ + public function processFile($filePath) + { + $messageHandler = Mage::getSingleton('xmlimport/messageHandler'); + /* @var $productBuilder C4B_XmlImport_Model_Products_ProductBuilder */ + $productBuilder = Mage::getModel('xmlimport/products_productBuilder'); + $productNodePosition = 0; + $xmlReader = new XMLReader(); + $xmlReader->open($filePath); + $products = array(); + while($xmlReader->read()) + { + if($xmlReader->nodeType != XMLReader::ELEMENT || $xmlReader->name != self::XML_NODE_NAME_PRODUCT) + { + continue; + } + $productNodePosition++; + $productData = $productBuilder->getProductData( $xmlReader->expand() ); + if( count($productBuilder->getErrors()) > 0 ) + { + $messageHandler->addError("Product at position {$productNodePosition} has errors:"); + } + if($productData == null) + { + $messageHandler->addError('Product will not be imported'); + } else + { + foreach ($productData as $productDataRow) + { + $products[] = $productDataRow; + } + } + $messageHandler->addErrorsForFile( basename($filePath), $productNodePosition, $productBuilder->getErrors() ); + } + + return $products; + } +} \ No newline at end of file diff --git a/src/app/code/community/C4B/XmlImport/Model/Products/ComplexAttribute.php b/src/app/code/community/C4B/XmlImport/Model/Products/ComplexAttribute.php new file mode 100644 index 0000000..6818df0 --- /dev/null +++ b/src/app/code/community/C4B/XmlImport/Model/Products/ComplexAttribute.php @@ -0,0 +1,90 @@ + xml node. It is responsible for extracting and formating data to an array element(s). + * + * @category C4B + * @package C4B_XmlImport + * @license http://opensource.org/licenses/osl-3.0.php Open Software Licence 3.0 (OSL-3.0) + * @author Dominik Meglic + * @copyright code4business Software GmbH + */ +class C4B_XmlImport_Model_Products_ComplexAttribute +{ + /** + * get complex attribute value from the given node. + * @param SimpleXMLElement $complexDataXmlNode + * @return null|array + */ + public function getComplexAttributeData($complexDataXmlNode) + { + if( !$this->_validateStructure($complexDataXmlNode) ) + { + return null; + } + return $this->_processXmlNode($complexDataXmlNode); + } + + /** + * Process the xml node, extract and format values from it and return it. + * @param SimpleXMLElement $complexDataXmlNode + * @return array + */ + protected function _processXmlNode($complexDataXmlNode) + { + $attributeData = array(); + /* @var $enumItem SimpleXMLElement */ + foreach ($complexDataXmlNode->children() as $enumItem) + { + foreach($enumItem as $attributeValue) + { + $value = trim(Mage::helper('xmlimport')->parseNumericEntities( $attributeValue->__toString() )); + if(array_key_exists($attributeValue->getName(), $attributeData)) + { + array_push($attributeData[$attributeValue->getName()], $value); + } else + { + $attributeData[$attributeValue->getName()] = array($value); + } + } + } + return $attributeData; + } + + /** + * Validate if children have same number of children. + * @param SimpleXMLElement $complexDataXmlNode + * @return boolean + */ + protected function _validateStructure($complexDataXmlNode) + { + if($complexDataXmlNode->count() == 0) + { + return true; + } + $expectedChildren = null; + $expectedChildrenCount = 0; + foreach ($complexDataXmlNode->children() as $enumItem) + { + if($expectedChildren == null) + { + /* @var $enumItemChild SimpleXMLElement */ + foreach($enumItem->children() as $nodeName => $enumItemChild) + { + $expectedChildrenCount++; + $expectedChildren[$nodeName] = true; + } + continue; + } + if($enumItem->count() != $expectedChildrenCount) + { + return false; + } + $compareFrom = Mage::helper('xmlimport')->xmlChildrenToArray($enumItem); + if(count(array_diff_key($compareFrom, $expectedChildren)) != 0) + { + return false; + } + } + return true; + } +} \ No newline at end of file diff --git a/src/app/code/community/C4B/XmlImport/Model/Products/ProductBuilder.php b/src/app/code/community/C4B/XmlImport/Model/Products/ProductBuilder.php new file mode 100644 index 0000000..cb26f37 --- /dev/null +++ b/src/app/code/community/C4B/XmlImport/Model/Products/ProductBuilder.php @@ -0,0 +1,289 @@ + + * @copyright code4business Software GmbH + */ +class C4B_XmlImport_Model_Products_ProductBuilder +{ + const EVENT_NAME_AFTER_SIMPLE_DATA = 'c4b_xmlimport_after_simple_data'; + const EVENT_NAME_AFTER_COMPLEX_DATA = 'c4b_xmlimport_after_complex_data'; + + protected $_errors = array(); + + /** + * Validate and extract product data from a single node. Returns an array of product data or null if errors occured. + * @param DOMNode $domNode + * @return array|null + */ + public function getProductData($domNode) + { + $doc = new DOMDocument('1.', 'UTF-8'); + $node = $doc->importNode($domNode, true); + $doc->appendChild( $node ); + + $xmlProductNode = simplexml_import_dom( $node ); + + $this->_errors = array(); + if( !property_exists( $xmlProductNode, 'simple_data' ) && !property_exists($xmlProductNode, 'stores') ) + { + $this->_errors[] = 'Missing or node.'; + return null; + } + + $productData = $this->_extractSimpleData($xmlProductNode, $this->_extractStoreCodes($xmlProductNode) ); + if($productData == null) + { + $this->_errors[] = 'Invalid simple data.'; + return null; + } + $productData = $this->_afterSimpleData($productData); + if($productData == null) + { + return null; + } + + $complexAttributeData = $this->_extractComplexData( $xmlProductNode ); + if($complexAttributeData == null) + { + $this->_errors[] = 'Invalid complex data.'; + return null; + } + $complexAttributeData = $this->_afterComplexData($complexAttributeData); + + if( !is_null($complexAttributeData) && !empty($complexAttributeData) ) + { + $productData['default'] = array_merge($productData['default'], $complexAttributeData); + } + + return $this->_formatData($productData); + } + + /** + * Return all error messages that were loged. + * @return multitype: + */ + public function getErrors() + { + return $this->_errors; + } + + /** + * Reformat data to non-asociative array. + * @return array + */ + protected function _formatData($productData) + { + $productStoreConfigs = array(); + + foreach($productData as $store => $productStoreConfig) + { + $productStoreConfigs[] = $productStoreConfig; + } + + return $productStoreConfigs; + } + + /** + * Get store view codes used in the XML and uses only the ones registered in the system + * @param SimpleXMLElement $xmlProductNode + * @return C4B_XmlImport_Model_Products_ProductBuilder; + */ + protected function _extractStoreCodes( $xmlProductNode ) + { + $systemStores = Mage::getModel('core/store')->getCollection(); + $systemStoreCodes = array('default'); + foreach ($systemStores as $singleStore) { + $systemStoreCodes[] = $singleStore->getCode(); + } + $productData['default'] = array(); + + foreach($xmlProductNode->stores->children() as $storeNode){ + $xmlStoreCode = $storeNode[0]->__toString(); + if(in_array($xmlStoreCode, $systemStoreCodes)) { + $productData[$xmlStoreCode] = Array(); + } + else { + $this->_errors[] = "The store '{$xmlStoreCode}' does not exist in the system. Data regarding this store will not be imported."; + } + } + + return $productData; + } + + /** + * * Extract simple_data values from currently loaded xml. + * @param SimpleXMLElement $xmlProductNode + * @param array $productData + * @return C4B_XmlImport_Model_Products_ProductBuilder + */ + protected function _extractSimpleData($xmlProductNode, $productData) + { + /* @var $attributeCreator C4B_XmlImport_Model_ProductCreator */ + $attributeCreator = Mage::getSingleton('xmlimport/attributeCreator'); + + if( !$xmlProductNode->simple_data->children() ) + { + return null; + } + /* @var $simpleAttributeNode SimpleXMLElement */ + foreach($xmlProductNode->simple_data->children() as $simpleAttributeNode) + { + foreach($productData as $store => $productStoreConfig) + { + if(!array_key_exists($store, $simpleAttributeNode)){ + continue; + } + + $value = $this->_parseValue($simpleAttributeNode->$store->__toString()); + if($value === false){ + continue; + } + + $importAttribute = $attributeCreator->createIfNotExists($simpleAttributeNode->getName()); + if($importAttribute){ + $productData[$store][$simpleAttributeNode->getName()] = $value; + } + } + } + return $productData; + } + + /** + * Parse the string to see if it contains a value. Returns false if the input string is empty + * @param string $stringValue + * @return boolean|string + */ + protected function _parseValue($stringValue) + { + $stringValue = trim($stringValue); + if( strlen($stringValue) === 0 ) + { + return false; + } + + return Mage::helper('xmlimport')->parseNumericEntities($stringValue); + } + + /** + * Processing after all attributes were collected. + * @param array $productData + * @return C4B_XmlImport_Model_Products_ProductBuilder + */ + protected function _afterSimpleData($productData) + { + foreach($productData as $storeCode => $storeSpecificData) + { + if(count($storeSpecificData) == 0) + { + unset($productData[$storeCode]); + continue; + } + if(!array_key_exists('sku', $storeSpecificData)) + { + $productData[$storeCode]['sku'] = null; + $productData[$storeCode]['_store'] = $storeCode; + } + } + + $transport = new Varien_Object(); + $transport->setData('product_data',$productData); + $transport->setData('errors',array()); + $transport->setData('invalidate_data',false); + + Mage::dispatchEvent(self::EVENT_NAME_AFTER_SIMPLE_DATA, array('transport' => $transport)); + $productData = $transport->getData('product_data'); + $this->_errors = array_merge( $this->_errors,$transport->getData('errors') ); + + if( $transport->getData('invalidate_data') == true ) + { + return null; + } + + return $productData; + } + + /** + * Extract complex data values from currently loaded xml node. + * @param SimpleXMLElement $xmlProductNode + * @return C4B_XmlImport_Model_Products_ProductBuilder + */ + protected function _extractComplexData($xmlProductNode) + { + if(property_exists($xmlProductNode, 'complex_data') && !$xmlProductNode->complex_data->children()) + { + return array(); + } + $productData = array(); + /* @var $complexAttribute C4B_XmlImport_Model_Products_ComplexAttribute */ + $complexAttribute = Mage::getModel('xmlimport/products_complexAttribute'); + /* @var $complexDataNode SimpleXMLElement */ + $complexAttributePosition = 0; + foreach($xmlProductNode->complex_data->children() as $complexDataNode) + { + $complexAttributePosition++; + $complexAttributeData = $complexAttribute->getComplexAttributeData($complexDataNode); + if( is_null($complexAttributeData) ) + { + $this->_errors[] = "Complex attribute at position {$complexAttributePosition} is invalid."; + return null; + } + $productData = array_merge($productData, $complexAttributeData); + } + return $productData; + } + + /** + * Additional altering of collected complex data. + * @param $productComplexData + * @return array + */ + protected function _afterComplexData($productComplexData) + { + $productComplexData = $this->_createNewCategories($productComplexData); + + $transport = new Varien_Object(); + $transport->setData('product_complex_data',$productComplexData); + $transport->setData('errors',array()); + $transport->setData('invalidate_data',false); + + Mage::dispatchEvent(self::EVENT_NAME_AFTER_COMPLEX_DATA,array('transport' => $transport)); + $this->_errors = array_merge( $this->_errors,$transport->getData('errors') ); + + if( $transport->getData('invalidate_data') == true ) + { + return null; + } + + return $transport->getData('product_complex_data'); + } + + /** + * Create new categories if configured to do so. + * @param array $productComplexData + * @return C4B_XmlImport_Model_Products_ProductBuilder + */ + protected function _createNewCategories($productComplexData) + { + if( !array_key_exists('_category', $productComplexData) + || Mage::getStoreConfig(C4B_XmlImport_Model_Importer::XML_PATH_PREPROCESSING_CREATE_CATEGORIES) == false ) + { + return $productComplexData; + } + /* @var $categoryCreator C4B_XmlImport_Model_CategoryCreator */ + $categoryCreator = Mage::getSingleton('xmlimport/categoryCreator'); + foreach( $productComplexData['_category'] as $key => $category ) + { + if( $categoryCreator->createIfItNotExists($category) == null ) + { + unset($productComplexData['_category'][$key]); + } + $this->_errors = array_merge( $this->_errors, $categoryCreator->getErrors() ); + } + return $productComplexData; + } +} diff --git a/src/app/code/community/C4B/XmlImport/Model/Resource/Helper/Mysql4.php b/src/app/code/community/C4B/XmlImport/Model/Resource/Helper/Mysql4.php new file mode 100644 index 0000000..0f21328 --- /dev/null +++ b/src/app/code/community/C4B/XmlImport/Model/Resource/Helper/Mysql4.php @@ -0,0 +1,34 @@ + + * @copyright code4business Software GmbH + */ +class C4B_XmlImport_Model_Resource_Helper_Mysql4 extends Mage_Core_Model_Resource_Helper_Mysql4 +{ + const LOCK_SETTING_TIMEOUT = '5'; + + /** + * Set named lock in DB + * @param string $lockName + * @return bool + */ + public function setNamedLock($lockName) + { + return (bool)$this->_getWriteAdapter()->query('SELECT GET_LOCK(?, ?);', array($lockName, self::LOCK_SETTING_TIMEOUT))->fetchColumn(); + } + + /** + * Release named lock in DB + * @param string $lockName + * @return bool + */ + public function releaseNamedLock($lockName) + { + return (bool)$this->_getWriteAdapter()->query('SELECT RELEASE_LOCK(?);', array($lockName))->fetchColumn(); + } +} \ No newline at end of file diff --git a/src/app/code/community/C4B/XmlImport/Model/Source/Attribute/ProcessingMode.php b/src/app/code/community/C4B/XmlImport/Model/Source/Attribute/ProcessingMode.php new file mode 100644 index 0000000..9202662 --- /dev/null +++ b/src/app/code/community/C4B/XmlImport/Model/Source/Attribute/ProcessingMode.php @@ -0,0 +1,26 @@ + + * @copyright code4business Software GmbH + */ +class C4B_XmlImport_Model_Source_Attribute_ProcessingMode +{ + const PROCESSING_MODE_INFORM_ONLY = 0; + const PROCESSING_MODE_CREATE_AND_INFORM = 1; + + /** + * Method that provides values for dropdown. + */ + public function toOptionArray() + { + return array( + array('value'=>self::PROCESSING_MODE_CREATE_AND_INFORM, 'label'=>Mage::helper('xmlimport')->__('Create missing attributes and send an e-mail')), + array('value'=>self::PROCESSING_MODE_INFORM_ONLY, 'label'=>Mage::helper('xmlimport')->__('Just send an e-mail with a list of missing attributes')), + ); + } +} diff --git a/src/app/code/community/C4B/XmlImport/Model/Uploader.php b/src/app/code/community/C4B/XmlImport/Model/Uploader.php new file mode 100644 index 0000000..940ecf9 --- /dev/null +++ b/src/app/code/community/C4B/XmlImport/Model/Uploader.php @@ -0,0 +1,66 @@ + + * @copyright code4business Software GmbH + */ +class C4B_XmlImport_Model_Uploader extends Mage_ImportExport_Model_Import_Uploader +{ + const XML_PATH_IMAGE_DUPLICATE_APPEND_SUFFIX = 'c4b_xmlimport/images/duplicate_append_suffix'; + + /** + * Check the configured behaviour for duplicate images. If set to no, images will not be renamed + * and will only be moved to media directory if it is newer. + * + * @return boolean + */ + protected function _isDuplicateAppendSuffix() + { + return Mage::getStoreConfig(self::XML_PATH_IMAGE_DUPLICATE_APPEND_SUFFIX); + } + + /** + * Vorbid renaming files to a new name if there already exists one image with the same name. + * @see Mage_ImportExport_Model_Import_Uploader::init() + */ + public function init() + { + parent::init(); + if( !$this->_isDuplicateAppendSuffix() ) + { + $this->setAllowRenameFiles(false); + } + } + + /** + * This overrides the original function and adds a check, that skips the import if the file + * in the media/catalog/product directory is newer than the one in media/import. + * @see Mage_ImportExport_Model_Import_Uploader::_moveFile() + */ + protected function _moveFile($tmpPath, $destPath) + { + if( !$this->_isDuplicateAppendSuffix()) + { + return parent::_moveFile($tmpPath, $destPath); + } + + $sourceFile = realpath($tmpPath); + $destinationFile = realpath($destPath); + + if (file_exists($destinationFile)) + { + $sourceStats = stat($sourceFile); + $destinationStats = stat($destinationFile); + if ($sourceStats['mtime'] <= $destinationStats['mtime']) + { + return true; + } + } + + return ($sourceFile !== false) ? copy($sourceFile, $destPath) : false; + } +} \ No newline at end of file diff --git a/src/app/code/community/C4B/XmlImport/doc/xml_structure.xml b/src/app/code/community/C4B/XmlImport/doc/xml_structure.xml new file mode 100644 index 0000000..0f11393 --- /dev/null +++ b/src/app/code/community/C4B/XmlImport/doc/xml_structure.xml @@ -0,0 +1,26 @@ + + + + + STORE_CODE + + + + + value + value + + + value + value + + + + + + default scope value + store scope value + + + + \ No newline at end of file diff --git a/src/app/code/community/C4B/XmlImport/etc/adminhtml.xml b/src/app/code/community/C4B/XmlImport/etc/adminhtml.xml new file mode 100644 index 0000000..0d8feea --- /dev/null +++ b/src/app/code/community/C4B/XmlImport/etc/adminhtml.xml @@ -0,0 +1,22 @@ + + + + + + + + + + + + XML Product Import + + + + + + + + + + \ No newline at end of file diff --git a/src/app/code/community/C4B/XmlImport/etc/config.xml b/src/app/code/community/C4B/XmlImport/etc/config.xml new file mode 100644 index 0000000..11ccbae --- /dev/null +++ b/src/app/code/community/C4B/XmlImport/etc/config.xml @@ -0,0 +1,63 @@ + + + + + 0.0.1 + + + + + + C4B_XmlImport_Block + + + + + C4B_XmlImport_Model + xmlimport_resource + + + C4B_XmlImport_Model_Resource + + + + C4B_XmlImport_Model_Import_Entity_Product + + + + + C4B_XmlImport_Model_Uploader + + + + + + C4B_XmlImport_Helper + + + + + + C4B_XmlImport + Mage_Core_Model_Resource_Setup + + + + + + + + media/import/XML + media/import/success + media/import/error + + + 1 + + + 1 + 0 + + + + \ No newline at end of file diff --git a/src/app/code/community/C4B/XmlImport/etc/system.xml b/src/app/code/community/C4B/XmlImport/etc/system.xml new file mode 100644 index 0000000..1a75f56 --- /dev/null +++ b/src/app/code/community/C4B/XmlImport/etc/system.xml @@ -0,0 +1,191 @@ + + + + + + 1000 + + + + + + c4b + text + 680 + 1 + 0 + 0 + + + + 10 + 1 + 0 + 0 + + + + text + 20 + 1 + 0 + 0 + This is the directory the product's import xml files are loaded from. + + + + text + 30 + 1 + 0 + 0 + Successfully imported product files are moved here. + + + + text + 40 + 1 + 0 + 0 + Failed import files are moved here. Check the log/import.log for error messages. + + + + + + 15 + 1 + 0 + 0 + + + + select + adminhtml/system_config_source_yesno + 5 + 1 + 0 + 0 + + + + + + + + + 20 + 1 + 0 + 0 + + + + select + adminhtml/system_config_source_category + 5 + 1 + 0 + 0 + + + + + + + select + adminhtml/system_config_source_yesno + 10 + 1 + 0 + 0 + + + + + + + select + xmlimport/source_attribute_processingMode + 20 + 1 + 0 + 0 + + + + textarea + 30 + 1 + 0 + 0 + + + + + + + + + 30 + 1 + 0 + 0 + + + + text + xmlimport/adminhtml_system_config_email + + + + adminhtml/system_config_backend_serialized_array + 10 + 1 + 0 + 0 + + + + select + adminhtml/system_config_source_email_template + + + + 15 + 1 + 0 + 0 + + + + text + xmlimport/adminhtml_system_config_email + + + + adminhtml/system_config_backend_serialized_array + 20 + 1 + 0 + 0 + + + + select + adminhtml/system_config_source_email_template + + + + 25 + 1 + 0 + 0 + + + + + + + \ No newline at end of file diff --git a/src/app/code/community/C4B/XmlImport/sql/xmlimport_setup/install-0.0.1.php b/src/app/code/community/C4B/XmlImport/sql/xmlimport_setup/install-0.0.1.php new file mode 100644 index 0000000..21b9cfc --- /dev/null +++ b/src/app/code/community/C4B/XmlImport/sql/xmlimport_setup/install-0.0.1.php @@ -0,0 +1,50 @@ + + * @copyright code4business Software GmbH + */ +/* @var $installer Mage_Core_Model_Resource_Setup */ +$installer = $this; + +$installer->startSetup(); + +$installer->setConfigData('fastsimpleimport/general/support_nested_arrays',true); + +/* @var $mailTemplate Mage_Core_Model_Email_Template */ +$errorMailTemplate = Mage::getModel('core/email_template')->loadByCode('Product Import Error Report'); +if( is_null($errorMailTemplate->getId()) ) +{ + $errorMailTemplate->addData(array( + 'template_code' => 'Product Import Error Report', + 'template_text' => '{{block type="xmlimport/importerror" result_data=$results start_time=$startTime}}', + 'template_type' => 2, //html + 'template_subject' => 'Errors during product import on {{var startTime}}', + 'template_styles' => 'body,td { color:#2f2f2f; font:11px/1.35em Verdana, Arial, Helvetica, sans-serif; }', + 'added_at' => time() + )); + $errorMailTemplate->save(); +} +$installer->setConfigData(C4B_XmlImport_Model_MessageHandler::XML_PATH_NOTIFICATIONS_IMPORT_ERROR_EMAIL_TEMPLATE, $errorMailTemplate->getId()); + +/* @var $mailTemplate Mage_Core_Model_Email_Template */ +$missingAttributesEmailTemplate = Mage::getModel('core/email_template')->loadByCode('Product Import Missing Attributes Report'); +if( is_null($missingAttributesEmailTemplate->getId()) ) +{ + $missingAttributesEmailTemplate->addData(array( + 'template_code' => 'Product Import Missing Attributes Report', + 'template_text' => '{{block type="xmlimport/missingattributes" attributes=$attributes attributes_created=$attributes_created}}', + 'template_type' => 2, //html + 'template_subject' => 'Missing attributes during product import on {{var startTime}}', + 'template_styles' => 'body,td { color:#2f2f2f; font:11px/1.35em Verdana, Arial, Helvetica, sans-serif; }', + 'added_at' => time() + )); + $missingAttributesEmailTemplate->save(); +} +$installer->setConfigData(C4B_XmlImport_Model_MessageHandler::XML_PATH_NOTIFICATIONS_MISSING_ATTRIBUTES_EMAIL_TEMPLATE, $missingAttributesEmailTemplate->getId()); + +$installer->endSetup(); \ No newline at end of file diff --git a/src/app/design/frontend/base/default/template/c4b/xmlimport/importreport.phtml b/src/app/design/frontend/base/default/template/c4b/xmlimport/importreport.phtml new file mode 100644 index 0000000..c16a768 --- /dev/null +++ b/src/app/design/frontend/base/default/template/c4b/xmlimport/importreport.phtml @@ -0,0 +1,43 @@ + + * @copyright code4business Software GmbH + */ + +/* @var $this C4B_XmlImport_Block_ImportError */ +?> +

Results from importing in

+ + + + + + +
Start time:getStartTime() ?>
Time taken:getTimeTaken(),3) ?> s(getTimeTaken() / 60) ,2) ?> min)
Memory used:getMemoryUsed() ?>
Number of imported files:getFileCount() ?>
Number of files with errors:getErrorFileCount() ?>
+
+getErrors() as $fileName => $productErrors):?> +
File:
+ + + + + + + $errors):?> + + $errorMessage): ?> + + + + + + + + +
Product positionError
+ \ No newline at end of file diff --git a/src/app/design/frontend/base/default/template/c4b/xmlimport/missing_attributes.phtml b/src/app/design/frontend/base/default/template/c4b/xmlimport/missing_attributes.phtml new file mode 100644 index 0000000..9f77d2f --- /dev/null +++ b/src/app/design/frontend/base/default/template/c4b/xmlimport/missing_attributes.phtml @@ -0,0 +1,27 @@ + + * @copyright code4business Software GmbH + */ + +/* @var $this C4B_XmlImport_Block_MissingAttributes */ +?> +

Missing attributes

+

+
+

Import file contains attributes that do not exist in the system

+
    +getAttributesList() as $missingAttribute):?> +
    • + +
+
+getAreAttributesCreated()):?> +

These attributes were created. For successful completion of importing please assign the attributes to the needed attribute sets + and run importing with the same file again.

+ \ No newline at end of file diff --git a/src/app/etc/modules/C4B_XmlImport.xml b/src/app/etc/modules/C4B_XmlImport.xml new file mode 100644 index 0000000..3af14c8 --- /dev/null +++ b/src/app/etc/modules/C4B_XmlImport.xml @@ -0,0 +1,12 @@ + + + + + true + community + + + + + + diff --git a/src/shell/xml_import.php b/src/shell/xml_import.php new file mode 100644 index 0000000..3ed4586 --- /dev/null +++ b/src/shell/xml_import.php @@ -0,0 +1,41 @@ + + * @copyright code4business Software GmbH + */ +class C4B_XmlImport_Importer extends Mage_Shell_Abstract { + + /** + * Start import process + * + */ + public function run() + { + /* @var $importer C4B_XmlImport_Model_Importer */ + $importer = Mage::getModel('xmlimport/importer'); + $importer->run(); + } + + /** + * Retrieve Usage Help Message + * + */ + public function usageHelp() + { + return <<run();