{"id":213,"date":"2012-08-17T14:38:33","date_gmt":"2012-08-17T21:38:33","guid":{"rendered":"http:\/\/www.lorrin.org\/blog\/?p=213"},"modified":"2013-03-25T22:59:32","modified_gmt":"2013-03-26T05:59:32","slug":"stripping-byte-order-mark","status":"publish","type":"post","link":"https:\/\/www.lorrin.org\/blog\/2012\/08\/17\/stripping-byte-order-mark\/","title":{"rendered":"Stripping Unicode Byte Order Mark to resolve SolrException: undefined field: “?myField” during Ingest"},"content":{"rendered":"

Solr<\/a> has a handy ability to ingest local CSV files<\/a>. The neatest aspect of which is that you can populate multi-valued fields by sub-parsing an individual field. E.g. the following will ingest \/tmp\/input.csv<\/tt> and split SomeField<\/tt> into multiple values by semi-colon delimiters:<\/p>\n

curl http:\/\/localhost:80\/solr\/my_core\/update\\?stream.file\\=\/tmp\/input.csv\\&stream.contentType\\=text\/csv\\;charset\\=utf-8\\&commit\\=true\\&f.SpmeField.split\\=true\\&f.SomeField.separator\\=%3B<\/pre>\n
When running an ingest, I got the following response, which was confusing since myField<\/tt> was, in fact, defined in my schema:<\/p>\n
<?xml version=\"1.0\" encoding=\"UTF-8\"?>\r\n<response>\r\n    <lst name=\"responseHeader\">\r\n        <int name=\"status\">400<\/int>\r\n        <int name=\"QTime\">1<\/int><\/lst>\r\n        <lst name=\"error\">\r\n            <str name=\"msg\">undefined field: \"myField\"<\/str>\r\n        <int name=\"code\">400<\/int>\r\n    <\/lst>\r\n<\/response><\/pre>\n
A peek in the log provided a clue (note the leading question mark):<\/p>\n
SEVERE: org.apache.solr.common.SolrException: undefined field: \"?myField\"<\/pre>\n
Examining a hex dump of the CSV file revealed that it started with a UTF-8 Byte Order Mark<\/a>:<\/p>\n
xxd \/tmp\/input.csv | head\r\n0000000: efbb bf...<\/pre>\n
One way to strip the BOM is with Bomstrip<\/a>, a collection of BOM-stripping implementations in various languages, including a Perl one-liner. Alternatively, just open the file in Vim, do :set nobomb<\/tt> and save<\/strong>. Done!<\/p>\n","protected":false},"excerpt":{"rendered":"
Solr has a handy ability to ingest local CSV files. The neatest aspect of which is that you can populate multi-valued fields by sub-parsing an individual field. E.g. the following will ingest \/tmp\/input.csv and split SomeField into multiple values by semi-colon delimiters: curl http:\/\/localhost:80\/solr\/my_core\/update\\?stream.file\\=\/tmp\/input.csv\\&stream.contentType\\=text\/csv\\;charset\\=utf-8\\&commit\\=true\\&f.SpmeField.split\\=true\\&f.SomeField.separator\\=%3B When running an ingest, I got the following response, which was […]<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[48],"tags":[70,68,69,38],"_links":{"self":[{"href":"https:\/\/www.lorrin.org\/blog\/wp-json\/wp\/v2\/posts\/213"}],"collection":[{"href":"https:\/\/www.lorrin.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.lorrin.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.lorrin.org\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.lorrin.org\/blog\/wp-json\/wp\/v2\/comments?post=213"}],"version-history":[{"count":5,"href":"https:\/\/www.lorrin.org\/blog\/wp-json\/wp\/v2\/posts\/213\/revisions"}],"predecessor-version":[{"id":252,"href":"https:\/\/www.lorrin.org\/blog\/wp-json\/wp\/v2\/posts\/213\/revisions\/252"}],"wp:attachment":[{"href":"https:\/\/www.lorrin.org\/blog\/wp-json\/wp\/v2\/media?parent=213"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.lorrin.org\/blog\/wp-json\/wp\/v2\/categories?post=213"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.lorrin.org\/blog\/wp-json\/wp\/v2\/tags?post=213"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}