[ERROR] request#show (ActionView::Template::Error) "incompatible character encodings: UTF-8 and ASCII-8BIT" #5783

garethrees · 2020-07-02T09:42:52Z

An ActionView::Template::Error occurred in request#show:

incompatible character encodings: UTF-8 and ASCII-8BIT
lib/mail_handler/backends/mail_backend.rb:60:in `mail_from_outlook'

-------------------------------
Request:
-------------------------------

* URL : https://www.asktheeu.org/en/request/public_documents_related_to_disi_6
* HTTP Method: GET
* Parameters : {"controller"=>"request", "action"=>"show", "url_title"=>"public_documents_related_to_disi_6", "locale"=>"en"}
* Timestamp : 2020-07-02 11:05:38 +0200
* Rails root : /REDACTED/alaveteli-2020-06-18T09-21-33

-------------------------------
Session:
-------------------------------

* session id: [FILTERED]
* data: {"session_id"=>"REDACTED", "locale"=>"en"}

-------------------------------
Backtrace:
-------------------------------

lib/mail_handler/backends/mail_backend.rb:60:in `mail_from_outlook'
lib/mail_handler/backends/mail_backend.rb:188:in `decode_attached_part'
lib/mail_handler/backends/mail_backend.rb:241:in `expand_and_normalize_parts'
lib/mail_handler/backends/mail_backend.rb:216:in `block in expand_and_normalize_parts'
lib/mail_handler/backends/mail_backend.rb:216:in `expand_and_normalize_parts'
lib/mail_handler/backends/mail_backend.rb:284:in `get_attachment_leaves'
lib/mail_handler/backends/mail_backend.rb:355:in `get_attachment_attributes'
app/models/incoming_message.rb:536:in `extract_attachments!'
app/models/incoming_message.rb:98:in `block in parse_raw_email!'
app/models/incoming_message.rb:97:in `parse_raw_email!'
app/models/incoming_message.rb:178:in `mail_from'
app/models/incoming_message.rb:197:in `safe_mail_from'
app/models/incoming_message.rb:220:in `specific_from_name?'
app/views/request/_incoming_correspondence.html.erb:4:in `_app_views_request__incoming_correspondence_html_erb___4239059047440180258_69914119794420'
app/views/request/_correspondence.html.erb:4:in `_app_views_request__correspondence_html_erb__4133394796544516923_69914122355520'
app/views/request/show.html.erb:70:in `block in _app_views_request_show_html_erb___2337845117304762896_69914123240540'
app/views/request/show.html.erb:68:in `_app_views_request_show_html_erb___2337845117304762896_69914123240540'
app/controllers/request_controller.rb:133:in `block (3 levels) in show'
app/controllers/request_controller.rb:132:in `block in show'
lib/alaveteli_localization.rb:43:in `with_locale'
app/controllers/request_controller.rb:83:in `show'
app/controllers/application_controller.rb:115:in `record_memory'
lib/strip_empty_sessions.rb:13:in `call'

The text was updated successfully, but these errors were encountered:

garethrees · 2020-07-02T09:43:09Z

Started seeing a lot of these come in over the past few days.

gbp · 2020-07-08T16:49:19Z

Can't even get to the admin request#show page

Request ID: 8270
Incoming message ID: 26831

gbp · 2020-07-08T20:31:55Z

This issue is fixed by aquasync/ruby-msg#16 but this conflicts with our changes at mysociety/ruby-msg#3. I'm not really sure what our changes are doing, anyone able to provide context... 5 years later? @crowbot @garethrees ?

garethrees · 2020-07-20T15:27:48Z

Might be able to manually add the response as a temporary fix until we can fix the underlying problem.

gbp · 2020-07-21T11:47:25Z

Might be able to manually add the response as a temporary fix until we can fix the underlying problem.

Not able to follow these step as generating a officer upload URL also errors

gbp · 2020-07-21T12:29:03Z

I've attempted to do the officer upload via the console (by calling RequestMailer#fake_response) and while it looks like it worked it hasn't added a message to the request ¯\_(ツ)_/¯

The message in question is only an acknowledgement and there has since been another response to the FOI. To get this working for this case I have simply deleted the acknowledgement.

In fae72e5 we introduced a fallback for incompatible character encodings, but that only covered the case where Mapi::Mime#is_multipart? returns true. This is a slight variation for the other branch of the conditional. The error is being raised where we have a `@body` in a different encoding to the default string encoding. In this case, just always force the string to be the same encoding as `@body` so we don’t get an error. This passes the spec for the particular file we’re seeing problems with, but I’m not at all sure this is the right thing to do. Maybe we should always treat anything in mapi as ASCII, and then force to UTF-8 for display? [1] Fixes mysociety/alaveteli#5783 [1] aquasync#9 (comment)

garethrees · 2020-09-22T16:24:05Z

Just noting the relevant snippet for reproducing the error:

require 'mapi/msg'
content = File.read '/home/vagrant/tmp/28101-outlook-attachment.msg'
msg = Mapi::Msg.open(StringIO.new(content))
msg.to_mime.to_s

garethrees · 2020-09-25T12:28:45Z

https://yehudakatz.com/2010/05/05/ruby-1-9-encodings-a-primer-and-the-solution-for-rails/

gbp · 2020-09-28T16:05:03Z

This is also happening for:

Request URL: https://www.asktheeu.org/en/request/contrato_no_lc_01464044
Request ID: 8516
Incoming message ID: 27953

garethrees · 2020-09-30T10:24:25Z

http://tenderlovemaking.com/2020/01/13/guide-to-string-encoding-in-ruby.html

garethrees · 2020-10-07T08:52:47Z

Another case:

Request URL: https://www.asktheeu.org/en/alaveteli_pro/info_requests/vaccine_talks_2
Request ID: 8558
Incoming message ID: 28101
Raw Email ID: 28101

Sometimes we receive messages with outlook attachments that can't be parsed due to an issue in mapi [1]. There's a potential fix [2] but it conflicts [3] with an existing patch we apply [4]. This at least allows users to download the raw attachment, rather than us preventing the entire request from loading because we raise an exception. Part of #5783. [1] aquasync/ruby-msg#15 [2] aquasync/ruby-msg#16 [3] #5783 (comment) [4] mysociety/ruby-msg#3

Sometimes we receive messages with outlook attachments that can't be parsed due to an issue in mapi [1]. There's a potential fix [2] but it conflicts [3] with an existing patch we apply [4]. This at least allows users to download the raw attachment, rather than us preventing the entire request from loading because we raise an exception. It's not easy to include an attachment in the specs to replicate a real error case due to the complexity of removing PII, so I've just stubbed the call to `.open` as we don't care about the specifics in this spec. `script/handle-mail-replies` needs an explicit require as we minimise the load path for that script. Part of #5783. [1] aquasync/ruby-msg#15 [2] aquasync/ruby-msg#16 [3] #5783 (comment) [4] mysociety/ruby-msg#3

garethrees · 2020-11-11T15:27:02Z

We haven't fixed the underlying issue (#5783 (comment)), but we've made our parsing much more resilient to failure (#5920) and the message content in the original bug report is now visible.

Request URL: https://www.asktheeu.org/en/alaveteli_pro/info_requests/vaccine_talks_2
https://www.asktheeu.org/en/request/contrato_no_lc_01464044

In these cases, the .msg files are downloadable and openable in (presumably) Outlook.

garethrees added bug Breaks expected functionality x:eu f:request-analysis labels Jul 2, 2020

garethrees assigned gbp Jul 6, 2020

garethrees assigned garethrees and unassigned gbp Sep 22, 2020

garethrees mentioned this issue Oct 13, 2020

[#5783] Improve resilience to unparsable outlook attachments #5920

Merged

garethrees closed this as completed Nov 11, 2020

laurentS mentioned this issue Aug 14, 2024

request#show: incompatible character encodings: ASCII-8BIT and UTF-8 #8360

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ERROR] request#show (ActionView::Template::Error) "incompatible character encodings: UTF-8 and ASCII-8BIT" #5783

[ERROR] request#show (ActionView::Template::Error) "incompatible character encodings: UTF-8 and ASCII-8BIT" #5783

garethrees commented Jul 2, 2020

garethrees commented Jul 2, 2020

gbp commented Jul 8, 2020

gbp commented Jul 8, 2020

garethrees commented Jul 20, 2020

gbp commented Jul 21, 2020

gbp commented Jul 21, 2020 •

edited

Loading

garethrees commented Sep 22, 2020

garethrees commented Sep 25, 2020

gbp commented Sep 28, 2020 •

edited by garethrees

Loading

garethrees commented Sep 30, 2020

garethrees commented Oct 7, 2020 •

edited

Loading

garethrees commented Nov 11, 2020 •

edited

Loading

[ERROR] request#show (ActionView::Template::Error) "incompatible character encodings: UTF-8 and ASCII-8BIT" #5783

[ERROR] request#show (ActionView::Template::Error) "incompatible character encodings: UTF-8 and ASCII-8BIT" #5783

Comments

garethrees commented Jul 2, 2020

garethrees commented Jul 2, 2020

gbp commented Jul 8, 2020

gbp commented Jul 8, 2020

garethrees commented Jul 20, 2020

gbp commented Jul 21, 2020

gbp commented Jul 21, 2020 • edited Loading

garethrees commented Sep 22, 2020

garethrees commented Sep 25, 2020

gbp commented Sep 28, 2020 • edited by garethrees Loading

garethrees commented Sep 30, 2020

garethrees commented Oct 7, 2020 • edited Loading

garethrees commented Nov 11, 2020 • edited Loading

gbp commented Jul 21, 2020 •

edited

Loading

gbp commented Sep 28, 2020 •

edited by garethrees

Loading

garethrees commented Oct 7, 2020 •

edited

Loading

garethrees commented Nov 11, 2020 •

edited

Loading