|
|
This seems to be the bug mentioned here:
http://po-ru.com/diary/fixing-invalid-utf-8-in-ruby-revisited/
That explains that some versions of Iconv don't ignore
invalid characters when converting to UTF-8 even with
//IGNORE if that invalid character happens to be at the end
of the string. In fact, as Matthew Somerville pointed out,
with some versions of iconv (e.g. 1.14 on Mac OS, apparently)
it's necessary to add and remove more than one space at the end,
in case the first character of the byte sequence indicates a
long sequence. We add and remove 4 to be on the safe side.
|
