Skip to main content

How to Import large CSV file in Drupal with Batch API , Beginners Guide Part I

Recently i have worked on a project which has lots of (in lakhs) data, I have a CSV file which we need to import in database, suppose the CSV have arround 2 lacks data, so when i try to import it then ststem consums too much time and gives the timeout error and failed to upload, to resole these type of issue the Batch API comes in My mind,

Why use batch API,

Batch API makes a queue of operations and performs each operations one by one, so system did not give the timeout issue, and batch API also shows the status of uploading and remaning data,


Here i am assuming you have basic knowledge of drupal module development,


Step -1: create a module within your drupal application, Lets assume module name is dharmend, Now i am going to create a menu , so i have added hook_menu in module file
 <?php  
 /* implement hook_menu to create a new menu,  
 *  
 */  
 function dharmend_menu(){  
      $items['csvImporter'] = array(  
     'title' => t('Upload Data From CSV File'),  
     'access callback' => 'user_access',  
     //'access arguments' => array('Import CSV labour data'),  
     'page callback' => 'csvImporter_pagecallback',  
     'type' => MENU_NORMAL_ITEM,  
    );  
    return $items ;  
 }  
Step-2 Now declare the callback function of the menu,
 /*  
 * menu Callback function  
 */  
 function csvImporter_pagecallback(){  
   $module_path = drupal_get_path('module', 'dharmend');  
   $form = drupal_get_form('csvImporter_form');  
   $output .= drupal_render($form);  
   return $output;  
  }  
  function csvImporter_form() {  
   $form['#attributes'] = array(  
     'enctype' => 'multipart/form-data'  
   );  
   $form['csvfile'] = array(   
    '#title' => t('CSV File'),  
    '#type' => 'file',  
    '#description' => ($max_size = parse_size(ini_get('upload_max_filesize'))) ? t('Due to server restrictions, the <strong>maximum upload file size is !max_size</strong>. Files that exceed this size will be disregarded.', array('!max_size' => format_size($max_size))) : '',    
   ) ;  
   $form['submit'] = array(  
    '#type' => 'submit',  
    '#value' => t('CSV Import'),  
   ) ;  
   $form['#validate'] = array(  
    'csvImporter_validate_fileupload',  
    'csvImporter_form_validate',  
   ) ;  
   return $form ;  
  }  

So here you can see that first i have created a new form to upload the csv file ,


Step -3 Add validate function to validate and upload file on server,

 function csvImporter_form_validate(&$form, &$form_state) {  
   if ( isset( $form_state['values']['csvupload'] ) ) {  
    if ( $handle = fopen($form_state['values']['csvupload'], 'r') ) {  
     $line_count = 1 ;  
     $first = TRUE ;  
     $counter = 0;  
     while($line = fgetcsv($handle)){//if ($line = fgetcsv($handle)){ //while($line = fgetcsv($handle)){  
       //drupal_set_message('Total row cont on csv'.count($line).'');  
     }  
     fclose($handle);  
    }  
    else {  
     form_set_error('csvfile', t('Unable to read uploaded file !filepath', array('!filepath' => $form_state['values']['csvupload'])));  
    }  
   }  
  }  

Comments

  1. It's not working , when i visit the page 'csvImporter' in my project, gives error 404 page not found,
    Please help

    ReplyDelete
  2. After enable the module you have to clear the menu cache , The the menu will be worked .

    ReplyDelete
  3. Now It works as expected , Thanks
    Very Nice & helpfull Article

    ReplyDelete

Post a Comment

Popular posts from this blog

Drupal 8 : Link actions,Link menus,Link Tasks,Routings

Drupal 8 : Link actions,Link menus,Link Tasks,Routings Link actions Local actions have also been moved out of the hook_menu() system in Drupal 8 .Use actions to define local operations such as adding new items to an administrative list (menus, contact categories, etc). Local actions are defined in a YAML format, named after the module they are defined by. Such as menu_ui.links.action.yml for this example from menu_ui module: menu_ui.link_add:   route_name: menu_ui.link_add   title: 'Add link'   appears_on:     - menu_ui.menu_edit Here, menu_ui.link_add: It is the Unique name of the link action Most likely start with module name, route_name : Name of the route it means when click the link it redirect to this route, appears_on :  An array of route names for this action to be display on. Now how to know the Route name of any internal/external admin pages like below, By through the drupal console we achieve it, drupal debug:router...

Mysql Interview Questions

Current mysql version : 8 ( last was 5.7 ,  5.7 to directly 8 ) SHOW FULL PROCESSLIST is used to see all the query executing when refresh the site. 1 second == 1000 mili second, 0-500 ms responce time of any query is ok How to Read the MySQL Slow Query Log :   The MySQL slow query log is where the MySQL database server registers all queries that exceed a given threshold of execution time. This can often be a good starting place to see which queries are slowest and how often they are slow. MySQL on your server is configured to log all queries taking longer than 0.1 seconds. /var/log/mysql/mysql-slow.log Use EXPLAIN or EXPLAIN EXTENDED to explain the query how it is executed. MySQL  describe  or  ANALYZE  command shows the structure of the table. Best practice in respect of performance : 1 - always use index, 2 - index types , primary index and combined field index like fname & lname in one index not two index, 3 - one index sc...

Get The field values of node in Drupal 8

use Drupal \ node \ NodeInterface ; /** * Implements hook_ENTITY_TYPE_insert() for node entities. * * This tests saving a node on node insert. * * @see \Drupal\node\Tests\NodeSaveTest::testNodeSaveOnInsert() */ function node_test_node_insert ( NodeInterface $node ) { // Set the node title to the node ID and save. if ( $node - > getTitle ( ) == 'new' ) { $node - > setTitle ( 'Node ' . $node - > id ( ) ) ; $node - > setNewRevision ( FALSE ) ; $node - > save ( ) ; } } Now There is so many functions are there to get the values, For All the functions available visit the API code, https://api.drupal.org/api/drupal/core%21modules%21node%21src%21NodeInterface.php/interface/NodeInterface/8.2.x Some of as below, Node edit form, Drupal 8 Automatically Load the whole object no need to load the entity like below, if ($event->getFormId() == 'node_alexa_audio_clips_edit_form') { $node = \Drupal::ro...