Skip to main content

How to Import large CSV file in Drupal with Batch API , Beginners Guide Part I

Recently i have worked on a project which has lots of (in lakhs) data, I have a CSV file which we need to import in database, suppose the CSV have arround 2 lacks data, so when i try to import it then ststem consums too much time and gives the timeout error and failed to upload, to resole these type of issue the Batch API comes in My mind,

Why use batch API,

Batch API makes a queue of operations and performs each operations one by one, so system did not give the timeout issue, and batch API also shows the status of uploading and remaning data,


Here i am assuming you have basic knowledge of drupal module development,


Step -1: create a module within your drupal application, Lets assume module name is dharmend, Now i am going to create a menu , so i have added hook_menu in module file
 <?php  
 /* implement hook_menu to create a new menu,  
 *  
 */  
 function dharmend_menu(){  
      $items['csvImporter'] = array(  
     'title' => t('Upload Data From CSV File'),  
     'access callback' => 'user_access',  
     //'access arguments' => array('Import CSV labour data'),  
     'page callback' => 'csvImporter_pagecallback',  
     'type' => MENU_NORMAL_ITEM,  
    );  
    return $items ;  
 }  
Step-2 Now declare the callback function of the menu,
 /*  
 * menu Callback function  
 */  
 function csvImporter_pagecallback(){  
   $module_path = drupal_get_path('module', 'dharmend');  
   $form = drupal_get_form('csvImporter_form');  
   $output .= drupal_render($form);  
   return $output;  
  }  
  function csvImporter_form() {  
   $form['#attributes'] = array(  
     'enctype' => 'multipart/form-data'  
   );  
   $form['csvfile'] = array(   
    '#title' => t('CSV File'),  
    '#type' => 'file',  
    '#description' => ($max_size = parse_size(ini_get('upload_max_filesize'))) ? t('Due to server restrictions, the <strong>maximum upload file size is !max_size</strong>. Files that exceed this size will be disregarded.', array('!max_size' => format_size($max_size))) : '',    
   ) ;  
   $form['submit'] = array(  
    '#type' => 'submit',  
    '#value' => t('CSV Import'),  
   ) ;  
   $form['#validate'] = array(  
    'csvImporter_validate_fileupload',  
    'csvImporter_form_validate',  
   ) ;  
   return $form ;  
  }  

So here you can see that first i have created a new form to upload the csv file ,


Step -3 Add validate function to validate and upload file on server,

 function csvImporter_form_validate(&$form, &$form_state) {  
   if ( isset( $form_state['values']['csvupload'] ) ) {  
    if ( $handle = fopen($form_state['values']['csvupload'], 'r') ) {  
     $line_count = 1 ;  
     $first = TRUE ;  
     $counter = 0;  
     while($line = fgetcsv($handle)){//if ($line = fgetcsv($handle)){ //while($line = fgetcsv($handle)){  
       //drupal_set_message('Total row cont on csv'.count($line).'');  
     }  
     fclose($handle);  
    }  
    else {  
     form_set_error('csvfile', t('Unable to read uploaded file !filepath', array('!filepath' => $form_state['values']['csvupload'])));  
    }  
   }  
  }  

Comments

  1. It's not working , when i visit the page 'csvImporter' in my project, gives error 404 page not found,
    Please help

    ReplyDelete
  2. After enable the module you have to clear the menu cache , The the menu will be worked .

    ReplyDelete
  3. Now It works as expected , Thanks
    Very Nice & helpfull Article

    ReplyDelete

Post a Comment

Popular posts from this blog

How to span column of custom table in Drupal

If you want to span the column of custom drupal table like below image, Follow the below code to make the header of the table , <?php $header = array('S.N','District', array('data' => '<div class ="house">Household </div><br><span>Rural</span> <p>Urban</p>','colspan' => 2), array('data' => '<div class ="house">Members</div> <br><span>Rural</span> <p>Urban</p>','colspan' => 2), 'Duplicate/Failed Registration', array('data' => '<div class ="house">Pending De duplication </div><br><span>Rural</span> <p>Urban</p>','colspan' => 2), 'Non Un-organised Workers', 'SSID Generated', 'No. of Card Personlised', ); $rows[] = arra...

Drupal 8 : Link actions,Link menus,Link Tasks,Routings

Drupal 8 : Link actions,Link menus,Link Tasks,Routings Link actions Local actions have also been moved out of the hook_menu() system in Drupal 8 .Use actions to define local operations such as adding new items to an administrative list (menus, contact categories, etc). Local actions are defined in a YAML format, named after the module they are defined by. Such as menu_ui.links.action.yml for this example from menu_ui module: menu_ui.link_add:   route_name: menu_ui.link_add   title: 'Add link'   appears_on:     - menu_ui.menu_edit Here, menu_ui.link_add: It is the Unique name of the link action Most likely start with module name, route_name : Name of the route it means when click the link it redirect to this route, appears_on :  An array of route names for this action to be display on. Now how to know the Route name of any internal/external admin pages like below, By through the drupal console we achieve it, drupal debug:router...

Cache In Drupal

Drupal 8 core caching modules: The Internal Page Cache module: this caches pages for anonymous users in the database. Pages requested by anonymous users are stored the first time they are requested and then are reused for future visitors. The Internal Dynamic Page Cache module: This is a key feature that Drupal 7 did not have. Unlike the Internal Page module, the Dynamic Page module aims to speed up the site for both anonymous and logged-in users. How Cache Works in Drupal: There are two modules available in the drupal core 1 - Internal page cache: The Internal Page Cache module caches pages for anonymous users in the database. Pages requested by anonymous users are stored the first time they are requested and then are reused. Configuring the internal page cache :   From the performance page we can set the maximum time how long browsers and proxies may cache pages based on the Cache-Control header. 2 - Internal dynamic page cache: Drupal 8 provides the Dynamic Page Cache m...