Skip to main content

How to Import large CSV file in Drupal with Batch API , Beginners Guide Part I

Recently i have worked on a project which has lots of (in lakhs) data, I have a CSV file which we need to import in database, suppose the CSV have arround 2 lacks data, so when i try to import it then ststem consums too much time and gives the timeout error and failed to upload, to resole these type of issue the Batch API comes in My mind,

Why use batch API,

Batch API makes a queue of operations and performs each operations one by one, so system did not give the timeout issue, and batch API also shows the status of uploading and remaning data,


Here i am assuming you have basic knowledge of drupal module development,


Step -1: create a module within your drupal application, Lets assume module name is dharmend, Now i am going to create a menu , so i have added hook_menu in module file
 <?php  
 /* implement hook_menu to create a new menu,  
 *  
 */  
 function dharmend_menu(){  
      $items['csvImporter'] = array(  
     'title' => t('Upload Data From CSV File'),  
     'access callback' => 'user_access',  
     //'access arguments' => array('Import CSV labour data'),  
     'page callback' => 'csvImporter_pagecallback',  
     'type' => MENU_NORMAL_ITEM,  
    );  
    return $items ;  
 }  
Step-2 Now declare the callback function of the menu,
 /*  
 * menu Callback function  
 */  
 function csvImporter_pagecallback(){  
   $module_path = drupal_get_path('module', 'dharmend');  
   $form = drupal_get_form('csvImporter_form');  
   $output .= drupal_render($form);  
   return $output;  
  }  
  function csvImporter_form() {  
   $form['#attributes'] = array(  
     'enctype' => 'multipart/form-data'  
   );  
   $form['csvfile'] = array(   
    '#title' => t('CSV File'),  
    '#type' => 'file',  
    '#description' => ($max_size = parse_size(ini_get('upload_max_filesize'))) ? t('Due to server restrictions, the <strong>maximum upload file size is !max_size</strong>. Files that exceed this size will be disregarded.', array('!max_size' => format_size($max_size))) : '',    
   ) ;  
   $form['submit'] = array(  
    '#type' => 'submit',  
    '#value' => t('CSV Import'),  
   ) ;  
   $form['#validate'] = array(  
    'csvImporter_validate_fileupload',  
    'csvImporter_form_validate',  
   ) ;  
   return $form ;  
  }  

So here you can see that first i have created a new form to upload the csv file ,


Step -3 Add validate function to validate and upload file on server,

 function csvImporter_form_validate(&$form, &$form_state) {  
   if ( isset( $form_state['values']['csvupload'] ) ) {  
    if ( $handle = fopen($form_state['values']['csvupload'], 'r') ) {  
     $line_count = 1 ;  
     $first = TRUE ;  
     $counter = 0;  
     while($line = fgetcsv($handle)){//if ($line = fgetcsv($handle)){ //while($line = fgetcsv($handle)){  
       //drupal_set_message('Total row cont on csv'.count($line).'');  
     }  
     fclose($handle);  
    }  
    else {  
     form_set_error('csvfile', t('Unable to read uploaded file !filepath', array('!filepath' => $form_state['values']['csvupload'])));  
    }  
   }  
  }  

Comments

  1. It's not working , when i visit the page 'csvImporter' in my project, gives error 404 page not found,
    Please help

    ReplyDelete
  2. After enable the module you have to clear the menu cache , The the menu will be worked .

    ReplyDelete
  3. Now It works as expected , Thanks
    Very Nice & helpfull Article

    ReplyDelete

Post a Comment

Popular posts from this blog

How to span column of custom table in Drupal

If you want to span the column of custom drupal table like below image, Follow the below code to make the header of the table , <?php $header = array('S.N','District', array('data' => '<div class ="house">Household </div><br><span>Rural</span> <p>Urban</p>','colspan' => 2), array('data' => '<div class ="house">Members</div> <br><span>Rural</span> <p>Urban</p>','colspan' => 2), 'Duplicate/Failed Registration', array('data' => '<div class ="house">Pending De duplication </div><br><span>Rural</span> <p>Urban</p>','colspan' => 2), 'Non Un-organised Workers', 'SSID Generated', 'No. of Card Personlised', ); $rows[] = arra...

Drupal 8 : Link actions,Link menus,Link Tasks,Routings

Drupal 8 : Link actions,Link menus,Link Tasks,Routings Link actions Local actions have also been moved out of the hook_menu() system in Drupal 8 .Use actions to define local operations such as adding new items to an administrative list (menus, contact categories, etc). Local actions are defined in a YAML format, named after the module they are defined by. Such as menu_ui.links.action.yml for this example from menu_ui module: menu_ui.link_add:   route_name: menu_ui.link_add   title: 'Add link'   appears_on:     - menu_ui.menu_edit Here, menu_ui.link_add: It is the Unique name of the link action Most likely start with module name, route_name : Name of the route it means when click the link it redirect to this route, appears_on :  An array of route names for this action to be display on. Now how to know the Route name of any internal/external admin pages like below, By through the drupal console we achieve it, drupal debug:router...

Get The field values of node in Drupal 8

use Drupal \ node \ NodeInterface ; /** * Implements hook_ENTITY_TYPE_insert() for node entities. * * This tests saving a node on node insert. * * @see \Drupal\node\Tests\NodeSaveTest::testNodeSaveOnInsert() */ function node_test_node_insert ( NodeInterface $node ) { // Set the node title to the node ID and save. if ( $node - > getTitle ( ) == 'new' ) { $node - > setTitle ( 'Node ' . $node - > id ( ) ) ; $node - > setNewRevision ( FALSE ) ; $node - > save ( ) ; } } Now There is so many functions are there to get the values, For All the functions available visit the API code, https://api.drupal.org/api/drupal/core%21modules%21node%21src%21NodeInterface.php/interface/NodeInterface/8.2.x Some of as below, Node edit form, Drupal 8 Automatically Load the whole object no need to load the entity like below, if ($event->getFormId() == 'node_alexa_audio_clips_edit_form') { $node = \Drupal::ro...