I've just been trying to figure out why my CSV data file was not importing correctly for certain records - they looked file, Both LibreOffice and Excel (to a certain degree - it really struggles with UTF8), opened the CSV file, with data expected to be where it was. Checking in plain text editors, I wasn't seeing extraneous commas or errant double quotes anywhere. I just couldn't see what the problem was with the data.

Well, eventually I figured it out - the CsvBulkUploader actually splits the original file into files of 1000 lines - I hadn't realised that before as I've never had to poke too deeply into the importing process beyond any custom importers I'd created. Well, usually I think this wouldn't be an issue, except one of the fields in the original file is an address field, containing multiple lines of addresses, yes, separated by newlines.

So, what is happening is the CsvBulkUploader is splitting the files in the middle of some address fields, so when the last record in the current file is imported, only the data up to the point in the address field that is included. Then the next file is started and the remaining data is loaded, being totally in the wrong place (so the created DataObjects don't have the required data). After a few rows, the data sorts itself out again until the next split.

A lot of background info there, probably more than required. Anyway, simple question - I see there is

private static $lines = 1000;

. Now other than setting that to a rather large value, is there any other way to just prevent the CsvBulkUploader from splitting the original file? I guess the other option is just to override the splitFile method to return an array of one file - the original one.

Just wondering if there was a better way (other than getting the client to remove all of the new lines in the address field).

Thanks,
Jason.

Just an update - overriding splitFile and just returning an array containing the original file works for me, just wondering if that is the best way.