These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

Issues, Workarounds & Localization

 
  • Topic is locked indefinitely.
 

EVE API: broken umlauts and wrong/no decoding of wallet journal entries
Author
Previous Topic Next Topic
Lorelei Tsu
R. Danneskjold and Sons Repossessions
#1 - 2012-01-03 12:19:13 UTC  |  Edited by: Lorelei Tsu
Using the EVE API I have stumbled upon API wallet-journal-entry-responses such as this:

Quote:

˂eveapi version="2">
˂currentTime>2012-01-03 11:43:08˂/currentTime>
˂result>
˂rowset name="entries" key="refID" columns="date,refID,refTypeID,ownerName1,ownerID1,ownerName2,ownerID2,argName1,argID1,amount,balance,reason">
˂row date="2012-01-02 21:08:00" refID="5287750012" refTypeID="37" ownerName1="Sarum Industries" ownerID1="555550" ownerName2="Lorelei Tsu" ownerID2="555551" argName1="Lorelei Tsu" argID1="555551" amount="-0.10" balance="55.04" reason="DESC: "\xDCmlaut T\xE4st" "/>
˂/rowset>
˂/result>
˂cachedUntil>2012-01-03 12:09:38˂/cachedUntil>
˂/eveapi>


Please note the broken umlauts in attribute "reason". The field should read "DESC: Ümlaut Täst". Umlauts and most probably other foreign (*) characters are being malformed. The whole "row" tag is invalid, btw, due to the doubled quotation marks.

The content-type of the API response is "application/xml; charset=utf-8" which is perfectly fine. Umlauts (and most other "foreign" characters) can be represented in this charset.

  1. Therefore adding quotation marks to the 'reason' field of account withdrawals and deposits is not necessary.
  2. Field "reason" can remain in Unicode. (Assuming the document is converted to utf-8 correctly upon being sent as response.)


Therefore, please check following:

  1. You're storing that data in unicode-field in database.
  2. Or, if as bytestring, run a "field.decode('utf-8')" after fetching from DB to convert it to Unicode. Blink
  3. Please don't emit sequences such as "\xFF" if the corresponding chars fit into utf-8.


(*) "foreign" as in foreign to someone exclusively using latin-1 charset. The usual suspects, *Räuspern*. Blink

http://tsu.eve-connect.com/ — (in German)
Lorelei Tsu
R. Danneskjold and Sons Repossessions
#2 - 2012-01-03 14:01:43 UTC
A client-side workaround for developers using the EVE API:

"Python you use" wrote:

from codecs import unicode_escape_decode

# ...
ᅚ@classmethod
ᅚdef fix_reason_field(cls, reason):
ᅚᅚ# removes the trailing \n which CCP adds to that field
ᅚᅚreason = reason.rstrip()
ᅚᅚif reason.startswith(u'DESC: "') and reason.endswith(u'"'):
ᅚᅚᅚreason = u'DESC: '+reason[7:-1]
ᅚᅚ# see https://forums.eveonline.com/default.aspx?g=posts&t=53350
ᅚᅚreason = unicode_escape_decode(reason)[0]
ᅚᅚreturn reason


That's still no fix for the database or API. Straight

http://tsu.eve-connect.com/ — (in German)