Make sure raw RABX column is utf8-encoded.

Without doing this, a call to e.g. $contact->set_extra_fields(@meta)
in PopulateServiceList.pm, with an unchanged meta that contains some
Unicode values, can write to the database (and cause an unneeded row
in the history table), because the column from the database is UTF-8
decoded, whilst the new text is UTF-8 encoded.

It looks like an attempt was made in filter_from_storage to fix this
issue, but the column comparison for marking a column as dirty takes
place without this being called.

1 files changed, 8 insertions, 4 deletions

diff --git a/perllib/FixMyStreet/DB/RABXColumn.pm b/perllib/FixMyStreet/DB/RABXColumn.pm
index 5f1583018..d14b48dc8 100644
--- a/perllib/FixMyStreet/DB/RABXColumn.pm
+++ b/perllib/FixMyStreet/DB/RABXColumn.pm
@@ -59,7 +59,6 @@ sub rabx_column {
                 my $self = shift;
                 my $ser  = shift;
                 return undef unless defined $ser;
-                utf8::encode($ser) if utf8::is_utf8($ser);
                 my $h = new IO::String($ser);
                 return RABX::wire_rd($h);
             },
@@ -78,18 +77,23 @@ sub rabx_column {
     $RABX_COLUMNS{ _get_class_identifier($class) }{$col} = 1;
 }
 
+# The underlying column should always be UTF-8 encoded bytes.
+sub get_column {
+    my ($self, $col) = @_;
+    my $res = $self->next::method ($col);
+    utf8::encode($res) if $RABX_COLUMNS{_get_class_identifier($self)}{$col} && utf8::is_utf8($res);
+    return $res;
+}
 
 sub set_filtered_column {
     my ($self, $col, $val) = @_;
 
-    my $class = ref $self;
-
     # because filtered objects may be expensive to marshall for storage there
     # is a cache that attempts to detect if they have changed or not. For us
     # this cache breaks things and our marshalling is cheap, so clear it when
     # trying set a column.
     delete $self->{_filtered_column}{$col}
-        if $RABX_COLUMNS{ _get_class_identifier($class) }{$col};
+        if $RABX_COLUMNS{ _get_class_identifier($self) }{$col};
 
     return $self->next::method($col, $val);
 }