MySQL and more

Thursday, March 12, 2009

LOAD DATA INFILE Gotcha

I was recently asked about errors in a LOAD DATA INFILE statement. Everything seemed straightforward enough except it was throwing warnings for some columns and not loading the correct values.

Given a simple set of data like this in a text file:
\N \N
18000 \N
18900 18900
546680 546680

and a table schema like this:

CREATE TABLE numbers (a int , b int );

Why would you get a result like this?
mysql> LOAD DATA INFILE 'c:\\test.txt' INTO TABLE numbers
Query OK, 4 rows affected, 2 warnings (0.00 sec)
Records: 4 Deleted: 0 Skipped: 0 Warnings: 0

mysql> show warnings;
+---------+------+-------------------------------------------------------+
| Level | Code | Message |
+---------+------+-------------------------------------------------------+
' for column 'b' at row 1 | integer value: 'N
' for column 'b' at row 2 | integer value: 'N
+---------+------+-------------------------------------------------------+
2 rows in set (0.00 sec)

mysql> SELECT * FROM numbers;
+--------+--------+
| a | b |
+--------+--------+
| NULL | 0 |
| 18000 | 0 |
| 18900 | 18900 |
| 546680 | 546680 |
+--------+--------+
4 rows in set (0.00 sec)


Notice it only freaked out on the NULL values and the end of the line!

The answer is one that's come up before, but for some reason I had to break out the hex editor before I remembered: The file was created on a windows machine, and had \r\n at the end of each line instead of just \n.

The solution is simple:
LOAD DATA INFILE 'c:\\test.txt' INTO TABLE numbers LINES TERMINATED BY '\r\n';

3 comments:

arjenAU said...

Also, use MyISAM when using LOAD DATA.
If you get warnings you can use TRUNCATE TABLE to essentially "rollback" and try again.
If it's all good, you can use INSERT ... SELECT .. to copy everything across to InnoDB.

You don't want a potential very large rollback in InnoDB (it's very very slow) and with the above use of MySQL's capabilities, there's a perfect way to deal with it.

Scott said...

A function you might find handy:

http://forge.mysql.com/tools/tool.php?id=187

shantanu said...

Create a table without any indexes if you are loading a lot of records.
Alter table later to add required keys.

Followers