Skip to main content

How to Import large CSV file in Drupal with Batch API , Beginners Guide Part I

Recently i have worked on a project which has lots of (in lakhs) data, I have a CSV file which we need to import in database, suppose the CSV have arround 2 lacks data, so when i try to import it then ststem consums too much time and gives the timeout error and failed to upload, to resole these type of issue the Batch API comes in My mind,

Why use batch API,

Batch API makes a queue of operations and performs each operations one by one, so system did not give the timeout issue, and batch API also shows the status of uploading and remaning data,


Here i am assuming you have basic knowledge of drupal module development,


Step -1: create a module within your drupal application, Lets assume module name is dharmend, Now i am going to create a menu , so i have added hook_menu in module file
 <?php  
 /* implement hook_menu to create a new menu,  
 *  
 */  
 function dharmend_menu(){  
      $items['csvImporter'] = array(  
     'title' => t('Upload Data From CSV File'),  
     'access callback' => 'user_access',  
     //'access arguments' => array('Import CSV labour data'),  
     'page callback' => 'csvImporter_pagecallback',  
     'type' => MENU_NORMAL_ITEM,  
    );  
    return $items ;  
 }  
Step-2 Now declare the callback function of the menu,
 /*  
 * menu Callback function  
 */  
 function csvImporter_pagecallback(){  
   $module_path = drupal_get_path('module', 'dharmend');  
   $form = drupal_get_form('csvImporter_form');  
   $output .= drupal_render($form);  
   return $output;  
  }  
  function csvImporter_form() {  
   $form['#attributes'] = array(  
     'enctype' => 'multipart/form-data'  
   );  
   $form['csvfile'] = array(   
    '#title' => t('CSV File'),  
    '#type' => 'file',  
    '#description' => ($max_size = parse_size(ini_get('upload_max_filesize'))) ? t('Due to server restrictions, the <strong>maximum upload file size is !max_size</strong>. Files that exceed this size will be disregarded.', array('!max_size' => format_size($max_size))) : '',    
   ) ;  
   $form['submit'] = array(  
    '#type' => 'submit',  
    '#value' => t('CSV Import'),  
   ) ;  
   $form['#validate'] = array(  
    'csvImporter_validate_fileupload',  
    'csvImporter_form_validate',  
   ) ;  
   return $form ;  
  }  

So here you can see that first i have created a new form to upload the csv file ,


Step -3 Add validate function to validate and upload file on server,

 function csvImporter_form_validate(&$form, &$form_state) {  
   if ( isset( $form_state['values']['csvupload'] ) ) {  
    if ( $handle = fopen($form_state['values']['csvupload'], 'r') ) {  
     $line_count = 1 ;  
     $first = TRUE ;  
     $counter = 0;  
     while($line = fgetcsv($handle)){//if ($line = fgetcsv($handle)){ //while($line = fgetcsv($handle)){  
       //drupal_set_message('Total row cont on csv'.count($line).'');  
     }  
     fclose($handle);  
    }  
    else {  
     form_set_error('csvfile', t('Unable to read uploaded file !filepath', array('!filepath' => $form_state['values']['csvupload'])));  
    }  
   }  
  }  

Comments

  1. It's not working , when i visit the page 'csvImporter' in my project, gives error 404 page not found,
    Please help

    ReplyDelete
  2. After enable the module you have to clear the menu cache , The the menu will be worked .

    ReplyDelete
  3. Now It works as expected , Thanks
    Very Nice & helpfull Article

    ReplyDelete

Post a Comment

Popular posts from this blog

How to span column of custom table in Drupal

If you want to span the column of custom drupal table like below image, Follow the below code to make the header of the table , <?php $header = array('S.N','District', array('data' => '<div class ="house">Household </div><br><span>Rural</span> <p>Urban</p>','colspan' => 2), array('data' => '<div class ="house">Members</div> <br><span>Rural</span> <p>Urban</p>','colspan' => 2), 'Duplicate/Failed Registration', array('data' => '<div class ="house">Pending De duplication </div><br><span>Rural</span> <p>Urban</p>','colspan' => 2), 'Non Un-organised Workers', 'SSID Generated', 'No. of Card Personlised', ); $rows[] = arra...

Get The field values of node in Drupal 8

use Drupal \ node \ NodeInterface ; /** * Implements hook_ENTITY_TYPE_insert() for node entities. * * This tests saving a node on node insert. * * @see \Drupal\node\Tests\NodeSaveTest::testNodeSaveOnInsert() */ function node_test_node_insert ( NodeInterface $node ) { // Set the node title to the node ID and save. if ( $node - > getTitle ( ) == 'new' ) { $node - > setTitle ( 'Node ' . $node - > id ( ) ) ; $node - > setNewRevision ( FALSE ) ; $node - > save ( ) ; } } Now There is so many functions are there to get the values, For All the functions available visit the API code, https://api.drupal.org/api/drupal/core%21modules%21node%21src%21NodeInterface.php/interface/NodeInterface/8.2.x Some of as below, Node edit form, Drupal 8 Automatically Load the whole object no need to load the entity like below, if ($event->getFormId() == 'node_alexa_audio_clips_edit_form') { $node = \Drupal::ro...

Dependency Injection in Drupal 8

Here are the important nuggets: DI is a design pattern used in programming. DI uses composition. DI achieves inversion of control. Dependency == service that your class needs == object of a certain type. Inject == provide == compose == assemble. Container == service container == dependency container. Instead of using  \Drupal::service('foo_service') , get the service from the  $container  if using a class. And the important reasons: Externalizing dependencies makes code easier to test. It allows dependencies to be replaced without interfering with other functionality. Retrieving dependencies from the container is better for performance. Services: node.grant_storage The easiest examples to find are services that have arguments, because you can search *.services.yml files for the word "arguments". In  node.services.yml  for example, there is this entry: node.grant_storage: class: Drupal\node\NodeGrantDatabaseStorage argument...