Retrofit + RxJava: Malformed Json Exception

nprugne · October 2, 2018, 12:42pm

Hi everyone!

I am using MOE to develop a sport teams management app on both Android and iOS. Under the hood, the app has a “common” layer which handles all the application logic. In this layer, I use the retrofit http client to get data from my server and the data are formatted in Json. So I have the following dependencies in my build.gradle file:

compile:'com.squareup.retrofit2:retrofit:2.4.0'
compile:'com.squareup.retrofit2:converter-gson:2.4.0'
compile:'com.squareup.retrofit2:adapter-rxjava:2.4.0'
compile:'io.reactivex:rxjava:1.3.8'
compile:'com.google.code.gson:gson:2.8.5'

Every thing works normally expect that sometimes when the Json is deserialized on iOS only, I get the following error message:

com.google.gson.stream.MalformedJsonException: Unterminated string at line 1 column 1025 path $.posts[1].text

On Android, the same data are perfectly deserialized and this error never occurs…

I have read tons of topics on the subject and I can’t figure out what the problem is! I have checked my Json with a validator and it is ok. Here is the problematic piece of Json if you want to check wall_chunk.json (2.7 KB). It has been retrieved with Postman not by the iOS app.

So I suppose that the real problem does not come from my Json. I think that the data are not fully retrieved by the app or truncated during the network call. Does somebody has ever faced the same problem ? This is a very critic bug some of my users can’t see their data anymore and I can’t do anything for them…

Best regards,
Nicolas

kisg · October 2, 2018, 9:17pm

Hi,

my suggestion is to somehow try to get the raw data that is received by the app and compare it with the raw data received on Android. I assume there will be some differences visible that could provide you with a clue. E.g. you could add some code to record the raw data received from the network using the OkHttp logging interceptor, and if there is a parsing error, you could record it, and provide a way to the user to send it to you (or even send it automatically using some kind of telemetry service).

It would be really great if you could keep us in the loop to see how this turns out.

Best Regards,
Gergely

nprugne · October 3, 2018, 9:40am

Thanks for the reply!

As you suggested, we have added some output logs to see the raw data retrieved by the client. To do so, we have used the HttpLoggingInterceptor of OkHttp. Here is the result on Android:

2018-10-03 10:43:27.895 11689-13215/com.digitalplumecompany.boostyourteam D/OkHttp: --> GET https://www.fakedomain.fr/teams/149/wall http/1.1
2018-10-03 10:43:27.895 11689-13215/com.digitalplumecompany.boostyourteam D/OkHttp: Accept: application/json
2018-10-03 10:43:27.895 11689-13215/com.digitalplumecompany.boostyourteam D/OkHttp: --> END GET
2018-10-03 10:43:28.175 11689-13215/com.digitalplumecompany.boostyourteam D/OkHttp: <-- 200 OK https://www.fakedomain.fr/teams/149/wall (279ms)
2018-10-03 10:43:28.175 11689-13215/com.digitalplumecompany.boostyourteam D/OkHttp: X-Powered-By: Express
2018-10-03 10:43:28.176 11689-13215/com.digitalplumecompany.boostyourteam D/OkHttp: Content-Type: application/json; charset=utf-8
2018-10-03 10:43:28.176 11689-13215/com.digitalplumecompany.boostyourteam D/OkHttp: Content-Length: 3074
2018-10-03 10:43:28.176 11689-13215/com.digitalplumecompany.boostyourteam D/OkHttp: ETag: W/"c02-6eopWb56C2G8DSYxhknQVPebo2s"
2018-10-03 10:43:28.176 11689-13215/com.digitalplumecompany.boostyourteam D/OkHttp: Date: Wed, 03 Oct 2018 08:43:29 GMT
2018-10-03 10:43:28.176 11689-13215/com.digitalplumecompany.boostyourteam D/OkHttp: Connection: close
2018-10-03 10:43:28.184 11689-13215/com.digitalplumecompany.boostyourteam D/OkHttp: {"posts":[{"comment":{"id":"7010","text":"\"Bonjour Dylanie. Ok pour Tess mais ton message d’origine a été supprimé. C’est bien pour « nice fête la rentrée » sur la prom pour la présentation de toutes les activités sportives et associatives de la ville de Nice ?!\"","elapsed_time":29041,"new":false,"player":{"id":"1","first_name":"Nicolas","last_name":"Prugne","profile_pic_url":"https://s3.eu-west-3.amazonaws.com/fakedomain-bucket/1f18af17-7000-446b-9c3c-8f8e841971e4.jpg"}},"id":"2721","text":"\"Arnaud dis à ta femme de faire des muffins 🤣🤣🤣\"","player":{"id":"1","first_name":"Nicolas","last_name":"Prugne","profile_pic_url":"https://s3.eu-west-3.amazonaws.com/fakedomain-bucket/1f18af17-7000-446b-9c3c-8f8e841971e4.jpg"},"team_id":"149","elapsed_time":29232,"new":false,"total_players_count":2,"total_views_count":2,"total_likes_count":0,"total_dislikes_count":0,"is_liked":false,"is_disliked":false,"hidden_comments_count":0},{"id":"2720","text":"\"Merci Dylanie ! Tess a rejoint le groupe 😉. Bonne soirée à tous\"","player":{"id":"1","first_name":"Nicolas","last_name":"Prugne","profile_pic_url":"https://s3.eu-west-3.amazonaws.com/fakedomain-bucket/1f18af17-7000-446b-9c3c-8f8e841971e4.jpg"},"team_id":"149","elapsed_time":29232,"new":false,"total_players_count":2,"total_views_count":2,"total_likes_count":0,"total_dislikes_count":0,"is_liked":false,"is_disliked":false,"hidden_comments_count":0},{"id":"2719","text":"\"Bonne semaine d’entraînement les filles ! Vous avez bien bosser.\n \nReposez vous\nA mardi 😊\"","player":{"id":"1","first_name":"Nicolas","last_name":"Prugne","profile_pic_url":"https://s3.eu-west-3.amazonaws.com/fakedomain-bucket/1f18af17-7000-446b-9c3c-8f8e841971e4.jpg"},"team_id":"149","elapsed_time":29232,"new":false,"total_players_count":2,"total_views_count":2,"total_likes_count":0,"total_dislikes_count":0,"is_liked":false,"is_disliked":false,"hidden_comments_count":0},{"id":"2718","text":"\"Nathhhhhhhhhh ton mari est arrivé et y a pas les muffinsssss 😢😢😢😢😢😢\"","player":{"id":"1","first_name":"Nicolas","last_name":"Prugne","profile_pic_url":"https://s3.eu-west-3.amazonaws.com/fakedomain-bucket/1f18af17-7000-446b-9c3c-8f8e841971e4.jpg"},"team_id":"149","elapsed_time":29232,"new":false,"total_players_count":2,"total_views_count":2,"total_likes_count":0,"total_dislikes_count":0,"is_liked":false,"is_disliked":false,"hidden_comments_count":0},{"id":"2717","text":"Cc dylanie ok pour samedi pour Juliette merci de me\nConfirmer l’adresse exacte et la tenue à mettre bonne journée","player":{"id":"1","first_name":"Nicolas","last_name":"Prugne","profile_pic_url":"https://s3.eu-west-3.amazonaws.com/fakedomain-bucket/1f18af17-7000-446b-9c3c-8f8e841971e4.jpg"},"team_id":"149","elapsed_time":29233,"new":false,"total_players_count":2,"total_views_count":2,"total_likes_count":0,"total_dislikes_count":0,"is_liked":false,"is_disliked":false,"hidden_comments_count":0}],"continuation_token":"1536838254_2717"}
2018-10-03 10:43:28.185 11689-13215/com.digitalplumecompany.boostyourteam D/OkHttp: <-- END HTTP (3074-byte body)

And here is what we have on iOS:

OkHttp 3 --> GET https://www.fakedomain.fr/teams/149/wall http/1.1
OkHttp 3 Accept: application/json
OkHttp 3 Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ8.eyJpZCI7IjIiLCJpYXQiOjE1Mzg1NTYwMjUsImV4cCI6MTUzODU1NjMyNX0.W8bzcKXn65uoxOaQwRdCKs012qtMovZQa9viYUgxfPk
OkHttp 3 --> END GET
OkHttp 3 <-- 200 OK https://www.fakedomain.fr/teams/149/wall (462ms)
OkHttp 3 X-Powered-By: Express
OkHttp 3 Content-Type: application/json; charset=utf-8
OkHttp 3 Content-Length: 3074
OkHttp 3 ETag: W/"c02-nvirdGPzt5DGWAuSXYHQPFFC3D8"
OkHttp 3 Date: Wed, 03 Oct 2018 08:41:08 GMT
OkHttp 3 {"posts":[{"comment":{"id":"7010","text":"\"Bonjour Dylanie. Ok pour Tess mais ton message d’origine a été supprimé. C’est bien pour « nice fête la rentrée » sur la prom pour la présentation de toutes les activités sportives et associatives de la ville de Nice ?!\"","elapsed_time":29039,"new":false,"player":{"id":"1","first_name":"Nicolas","last_name":"Prugne","profile_pic_url":"https://s3.eu-west-3.amazonaws.com/fakedomain-bucket/1f18af17-7000-446b-9c3c-8f8e841971e4.jpg"}},"id":"2721","text":"\"Arnaud dis à ta femme de faire des muffins 🤣🤣🤣\"","player":{"id":"1","first_name":"Nicolas","last_name":"Prugne","profile_pic_url":"https://s3.eu-west-3.amazonaws.com/fakedomain-bucket/1f18af17-7000-446b-9c3c-8f8e841971e4.jpg"},"team_id":"149","elapsed_time":29230,"new":false,"total_players_count":2,"total_views_count":2,"total_likes_count":0,"total_dislikes_count":0,"is_liked":false,"is_disliked":false,"hidden_comments_count":0},{"id":"2720","text":"\"Merci Dylanie ! Tess a rejoint le groupe 😉. Bonne soirée à tous\"","player":{"id":"1","first_name":"Nicolas","last_name":"Prugne","profile_pic_url":"https://s3.eu-west-3.amazonaws.com/fakedomain-bucket/1f18af17-7000-446b-9c3c-8f8e841971e4.jpg"},"team_id":"149","elapsed_time":29230,"new":false,"total_players_count":2,"total_views_count":2,"total_likes_count":0,"total_dislikes_count":0,"is_liked":false,"is_disliked":false,"hidden_comments_count":0},{"id":"2719","text":"\"Bonne semaine d’entraînement les filles ! Vous avez bien bosser.\n \nReposez vous\nA mardi 😊\"","player":{"id":"1","first_name":"Nicolas","last_name":"Prugne","profile_pic_url":"https://s3.eu-west-3.amazonaws.com/fakedomain-bucket/1f18af17-7000-446b-9c3c-8f8e841971e4.jpg"},"team_id":"149","elapsed_time":29230,"new":false,"total_players_count":2,"total_views_count":2,"total_likes_count":0,"total_dislikes_count":0,"is_liked":false,"is_disliked":false,"hidden_comments_count":0},{"id":"2718","text":"\"Nathhhhhhhhhh ton mari est arrivé et y a pas les muffinsssss 😢😢😢😢😢😢\"","player":{"id":"1","first_name":"Nicolas","last_name":"Prugne","profile_pic_url":"https://s3.eu-west-3.amazonaws.com/fakedomain-bucket/1f18af17-7000-446b-9c3c-8f8e841971e4.jpg"},"team_id":"149","elapsed_time":29230,"new":false,"total_players_count":2,"total_views_count":2,"total_likes_count":0,"total_dislikes_count":0,"is_liked":false,"is_disliked":false,"hidden_comments_count":0},{"id":"2717","text":"Cc dylanie ok pour samedi pour Juliette merci de me\nConfirmer l’adresse exacte et la tenue à mettre bonne journée","player":{"id":"1","first_name":"Nicolas","last_name":"Prugne","profile_pic_url":"https://s3.eu-west-3.amazonaws.com/fakedomain-bucket/1f18af17-7000-446b-9c3c-8f8e841971e4.jpg"},"team_id":"149","elapsed_time":29231,"new":false,"total_players_count":2,"total_views_count":2,"total_likes_count":0,"total_dislikes_count":0,"is_liked":false,"is_disliked":false,"hidden_comments_count":0}],"continuation_token":"1536838254_2717"}
OkHttp 3 <-- END HTTP (3074-byte body)

We have used a tool to compare the raw data (this part {“posts”: [{ … }]}) that we call a “wall chunk” in our app and we have found no differences except the elapsed_time fields.

Here is the stack trace of the error:

stackTrace = {StackTraceElement[27]@2641} 
 0 = {StackTraceElement@2644} "com.google.gson.stream.JsonReader.syntaxError(JsonReader.java:1568)"
 1 = {StackTraceElement@2645} "com.google.gson.stream.JsonReader.nextQuotedValue(JsonReader.java:1031)"
 2 = {StackTraceElement@2646} "com.google.gson.stream.JsonReader.nextString(JsonReader.java:815)"
 3 = {StackTraceElement@2647} "com.google.gson.internal.bind.TypeAdapters$16.read(TypeAdapters.java:402)"
 4 = {StackTraceElement@2648} "com.google.gson.internal.bind.TypeAdapters$16.read(TypeAdapters.java:390)"
 5 = {StackTraceElement@2649} "com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:131)"
 6 = {StackTraceElement@2650} "com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:222)"
 7 = {StackTraceElement@2651} "com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:41)"
 8 = {StackTraceElement@2652} "com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:82)"
 9 = {StackTraceElement@2653} "com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:61)"
 10 = {StackTraceElement@2654} "com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:131)"
 11 = {StackTraceElement@2655} "com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:222)"
 12 = {StackTraceElement@2656} "retrofit2.converter.gson.GsonResponseBodyConverter.convert(GsonResponseBodyConverter.java:39)"
 13 = {StackTraceElement@2657} "retrofit2.converter.gson.GsonResponseBodyConverter.convert(GsonResponseBodyConverter.java:27)"
 14 = {StackTraceElement@2658} "retrofit2.ServiceMethod.toResponse(ServiceMethod.java:122)"
 15 = {StackTraceElement@2659} "retrofit2.OkHttpCall.parseResponse(OkHttpCall.java:217)"
 16 = {StackTraceElement@2660} "retrofit2.OkHttpCall.execute(OkHttpCall.java:180)"
 17 = {StackTraceElement@2661} "retrofit2.adapter.rxjava.CallExecuteOnSubscribe.call(CallExecuteOnSubscribe.java:40)"
 18 = {StackTraceElement@2662} "retrofit2.adapter.rxjava.CallExecuteOnSubscribe.call(CallExecuteOnSubscribe.java:24)"
 19 = {StackTraceElement@2663} "retrofit2.adapter.rxjava.BodyOnSubscribe.call(BodyOnSubscribe.java:36)"
 26 = {StackTraceElement@2670} "com.digitalplumecompany.boostyourteam.schedulers.ScheduledAction$1.call_addExecutionBlock(ScheduledAction.java:57)"
 25 = {StackTraceElement@2669} "rx.Single$13$1.call(Single.java:2051)"
 24 = {StackTraceElement@2668} "rx.Single.subscribe(Single.java:1979)"
 23 = {StackTraceElement@2667} "rx.internal.operators.OnSubscribeSingle.call(OnSubscribeSingle.java:27)"
 22 = {StackTraceElement@2666} "rx.internal.operators.OnSubscribeSingle.call(OnSubscribeSingle.java:81)"
 21 = {StackTraceElement@2665} "rx.Observable.unsafeSubscribe(Observable.java:10327)"
 20 = {StackTraceElement@2664} "retrofit2.adapter.rxjava.BodyOnSubscribe.call(BodyOnSubscribe.java:28)"

We still have the issue and we don’t know where does it come from.

Regards,
Nicolas

kisg · October 3, 2018, 6:40pm

Thanks for the update.

Ok, this is getting interesting. The good news is that there is no problem in the network code.

Could you create a self contained minimal sample app that just tries to parse this json file with the GsonResponseBodyConverter.java (or using Gson directly) and reproduces the problem?

Also, a workaround could be to try if you can use converter-jackson instead of converter-gson for retrofit. We have used Jackson in the past in MOE, so I am pretty sure that it should work fine.

Best Regards,
Gergely