Crash Reporting

I’d like to continue the discussion regarding Crash Reporting started on the old Intel forums here:

https://software.intel.com/en-us/forums/multi-os-engine/topic/667934

Gergely, you mentioned the following plan of attack:

  1. Basic Support: Make MOE work with the current “native” iOS crash reporting engines. This basically means that an uncaught exception on the Java side should be transformed to a native exception in a way that the Java stack trace gets recorded by the crash reporting frameworks (at least as a string message).
  1. Advanced Support: Provide a way to record Java and native exceptions as separate, distinguishable events, and in a form so the stack traces from both sides can be processed by the crash reporter service backend correctly (and provide the additional analytics functions that you mentioned). To do this, we will have to refactor the Android SDK (and possibly even the backend) for a chosen crash reporting service, so it can work with MOE as well. This first implementation will serve as a proof of concept for implementing support for other similar services.

So, where do we stand on this? Is step 1 as simple as creating a standard binding around the client libraries distributed by Fabric or Apteligent? Or do we need some internal changes to Multi-OS Engine first to support this?

Hi Paul,

we looked at the “Basic Support” scenario in more detail, and it looks like that it is not possible to customize the standard iOS crash report contents (i.e. to include the Java stacktrace). If there is a way to do it, I am very interested to find out how.

However, most Crash Reporting frameworks provide a way to include custom logs in crash reports. For example Fabric / Crashlytics includes the CSL_LOG() family of functions / macros. The good news is, that we can create a binding to these functions with Nat/J, and create a standard Java Unhandled Exception handler to log any unhandled exceptions. The same approach can be used with virtually every crash reporting framework to get the Java stacktrace into the reporting system as a simple log.

For those Crash Reporting frameworks, where the client side SDK is open source, the Android version can be customized to upload the Java stracktraces as if it came from the Android version (probably the app version has to be configured accordingly, so the iOS stacktraces appear as separate from the Android ones. At the same time the native side stack traces can be captured and uploaded by the iOS version of each crash reporting framework.

We will create a sample based on the above approach, once the open-sourcing is finished.

What do you think?

Best Regards,
Gergely

That sounds reasonable to me Gergely. I’ve been trying on my own end to figure out how to wrap up the Fabric libs so I can test using the CSL_LOG() method. I’m a bit confused though as Fabric is only available as a cocoapod and I’ve not used that system before. Should I be manually updating my xcode project to use cocoapods and installing the relevant pods outside of Android Studio, or should I rely entirely on the Android Studio plugin that wraps it all up in jar? Are there any good step-by-step instructions for using cocoapods with multi-os engine?

Paul

OK, well, I’ve got something working. I created the jar wrappers for Fabric and Crashlytics, which appear to contain the framework files, and I added to my build.gradle dependencies:

compile fileTree(dir: 'lib', include: ['*.jar'])

This made all the java code happy to compile, but I was still getting errors about the framework being missing. Finally I created a Podfile with those two pods listed, installed them, and then manually added the frameworks to my xcode project. That seems to have worked.

After digging a bit through the Crashlytics docs, I ended up with the following for my uncaught exception handler, which I hope will be a bit better than just logging the stack:

import com.intel.crashlytics.CLSStackFrame;
import com.intel.crashlytics.Crashlytics;

import ios.foundation.NSMutableArray;

public class IOSUncaughtExceptionHandler implements Thread.UncaughtExceptionHandler {

	private Thread.UncaughtExceptionHandler _defaultHandler;

	public IOSUncaughtExceptionHandler() {
		this._defaultHandler = Thread.getDefaultUncaughtExceptionHandler();
	}

	public void uncaughtException(Thread t, Throwable e) {
		NSMutableArray<CLSStackFrame> stack = (NSMutableArray<CLSStackFrame>)NSMutableArray.alloc().init();
		for (StackTraceElement element : e.getStackTrace()) {
			CLSStackFrame frame = CLSStackFrame.alloc().init();
			frame.setFileName(element.getFileName());
			frame.setLineNumber(element.getLineNumber());
			frame.setSymbol(element.getMethodName());
			stack.add(frame);
		}
		Crashlytics.sharedInstance().recordCustomExceptionNameReasonFrameArray(e.getClass().getName(), e.getMessage(), stack);

		this._defaultHandler.uncaughtException(t, e);
	}
}

I haven’t seen a crash actually show up in my dashboard yet. I’m hoping there’s just a delay in processing reports on the Fabric side. I’ll post more findings as they come.

Paul

1 Like

Dear Paul,

this is great, thank you!

Best Regards,
Gergely

No sweat, and I can report that it appears to be working pretty well. I know from experience that it can take Crashlytics a day or so to start actually showing data, so I tested the above and let it sit overnight. The reports are showing up now and look pretty good. One minor change I’ve made is to add the class name to the symbol thus:

frame.setSymbol(element.getClassName() + "." + element.getMethodName());

That helps the stack look a little more similar to those generated on the Android side. I’m also getting native crashes showing up thanks to just registering the library. They can be a little difficult to read, but are certainly better than nothing. Here’s an example:

Crashed: main
0  libobjc.A.dylib                0x24a5bdd8 objc_destructInstance + 15
1  libobjc.A.dylib                0x24a5be0f object_dispose + 14
2  libobjc.A.dylib                0x24a5be0f object_dispose + 14
3  ios-moe                        0xe48d7 -[ALBuffer dealloc] (ALBuffer.m:105)
4  libobjc.A.dylib                0x24a6cf8b objc_object::sidetable_release(bool) + 150
5  ios-moe                        0xef2f1 -[ALSource setBuffer:] (ALSource.m:217)
6  ios-moe                        0xf2ca3 -[ALSource play:gain:pitch:pan:loop:] (ALSource.m:1075)
7  ios-moe                        0xe81cd -[ALChannelSource play:gain:pitch:pan:loop:] (ALChannelSource.m:274)
8  ios-moe                        0xe41f3 -[OALSimpleAudio playBuffer:volume:pitch:pan:loop:] (OALSimpleAudio.m:691)
9  MOE                            0x372a01c ffi_call_SYSV + 28
10 MOE                            0x372c2b3 ffi_call + 18
11 MOE                            0x3725631 std::__1::function<void (unsigned int, _ffi_type**, void**)>::operator()(unsigned int, _ffi_type**, void**) const + 44
12 MOE                            0x3725561 ValueConverter<(ValueConverterKind)1>::ValueConverter(ValueConverter<(ValueConverterKind)1>::Descriptor const&, std::__1::function<void (unsigned int, _ffi_type**, void**)>) + 202
13 MOE                            0x372281f javaToNativeMessageHandler(ffi_cif*, void*, void**, void*) + 1286
14 MOE                            0x372c5e5 ffi_closure_inner_SYSV + 112
15 MOE                            0x372a0a4 ffi_closure_SYSV + 36

The one piece that’s still confusing to me is the manual inclusion of the frameworks into my project. Are they not supposed to be loaded out of the packaged jar file? Does the use of libgdx add an extra wrinkle here that would have otherwise been smoother was I just writing a vanilla multi-os-engine project?

1 Like

Dear Paul,
How did you create the jar wrappers for Fabric and Crashlytics ?
And How did you created a Podfile with those two pods listed, installed them, and then manually added the frameworks to your xcode project ?
How did you use IOSUncaughtExceptionHandler?
Help me, please!

I generated the jars for the pods using the documented method here:
https://doc.multi-os-engine.org/multi-os-engine/5_using_thirdparty/2_wrap_android/2_toxcode/3_UsingCocoapods/cocoapods.html?highlight=cocoapods

Getting the project to link to the framework was a bit annoying, as it didn’t appear that the project was discovering the frameworks embedded in the jar files. I ended up just creating a dummy XCode project to install the pods, copying the framework files over, then manually adding them to the libgdx generated xcode project. I think I read somewhere that all this is changing in MOE 1.3?

As for actually using the IOSUncaughtExceptionHandler, I added the following to my main IOSLauncher class:

    @Override
public boolean applicationDidFinishLaunchingWithOptions(UIApplication application, NSDictionary<?, ?> launchOptions) {
    boolean retVal = super.applicationDidFinishLaunchingWithOptions(application, launchOptions);
    if (retVal) {
        NSMutableArray array = NSMutableArray.alloc().init();
        array.add(Crashlytics.class_objc_static());
        Fabric.with(array);
        Thread.setDefaultUncaughtExceptionHandler(new IOSUncaughtExceptionHandler());
    }
    return retVal;
}

Hope that helps. Good luck.

2 Likes

I was also looking at this and wonder if you had a chance to update this to MOE 1.3-beta-2 yet? There is a new tool to generate java bindings.

On a side note, for Crashlytics, does it have to be a native wrapper? Or it would work to just use Crashlytics runtime for Java?

Hi Eugene,

Recently I have successfully connected Fabric and Crashlytics using the new Java Binding Generator. It’s working correctly, I have just one small issue with the Crashlytics raport - @PaulS custom ExceptionHandler is not working for me.

I’ve used the iOS native Fabric and Crashlytics pods:

  1. Install CocoaPods on your computer
sudo gem install cocoapods
  1. In terminal proceed to the xcode project directory and create new pod file
pod init
  1. Edit Podfile - use frameworks and set target to 9.0. Then save the file.
vim Podfile
platform :ios, 9.0
use_frameworks!
target 'MyTarget' do
  # Pods for MyTarget
   pod 'Fabric'
   pod 'Crashlytics'
end
  1. Force remove the xcode configurations for your targets
Project navigator > Project > Info > Configurations

Set all to ‘None
More on: http://stackoverflow.com/questions/26287103/cocoapods-warning-cocoapods-did-not-set-the-base-configuration-of-your-project/33509278#33509278
5) Quit xcode!
6) Ensure that xcode is not running!

Note: Running xcode will cause problems during pod installing. If you had your xcode running and you are having problems - try repeating this tutorial after removing all files modified by pod install.

  1. Install all pods from the Podfile - go to Podfile dir in terminal and type:
pod install
  1. Change build settings in xcode
Project navigator > Targets > MyTarget > Build Phases > Link Binary With Libraries
  1. Link your new pods. If you cannot find them try:
New > Add Other > ios > xcode > Pods > Fabric > iOS > Fabric.framework
  1. Some libraries are needed extra for Fabric and Crashlytics (if you won’t have them you will get later an error and then find the needed libraries in some StackOverflow answers :wink: ):

  2. Add your API key (after creating a Fabric account you will find it here: https://fabric.io/kits/ios/crashlytics/install)
    Remember to do all steps from Add a Run Script Build Phase and Add Your API Key.

  3. Add -ObjC to your linker flags:

Project > Targets > MyTarget > Build settings > Linking > Other Linker Flags

Double click and add new flag -ObjC

  1. If building through xcode is passing then proceed to Android Studio. Right click on project and do
MOE Actions > Create New Bindings
  1. Create .nbc file and proceed to form. In new window select Header and fill the form as below:
Name: <YOUR_POD_NAME>
Header path: xcode/Pods/<YOUR_POD_NAME>/<YOUR_POD_NAME>
(or another path to pod headers)
Base package name: com.yourapp.ios.pods.<YOUR_POD_NAME>
Import headers:
#import "<YOUR_POD_HEADER_FILE>.h"
(same as the objc import added in native pod usage)
  1. Click on gear icon and click Generate bindings. The IDE will now freeze (@kisg ;)) and after a while it should generate Java classes in com.yourapp.ios.pods.<YOUR_POD_NAME>

  2. In your main class make sure that you initialize them correctly:

    @Override
    public boolean applicationDidFinishLaunchingWithOptions(UIApplication application, NSDictionary launchOptions) {
        Fabric.with(NSArray.arrayWithObject(Crashlytics.class_objc_static()));
        return true;
    }

The only issue I have is that all my reports is grouped in a single one:

Which have different exceptions inside:

I’m not sure why the @PaulS custom exception handler is not working here.

Let me know if I can help you with any more issues!

5 Likes

Thank you Mateusz. It will take me some time to process all that. Great stuff.
Just one thing I’m still confused about. What you gaining by hooking up iOS version of Crashlytics vs. using Java one under MOE… Is it just to get the native stack traces?

To be honest - I thought that the only way to make Crashlytics work with iOS is to use the native version. As far as I know, there is no clean-java version of Crashlytics, there is only the Android version, which wouldn’t work with iOS - it requires a lot of Android dependencies:

Native iOS stack traces are also not so valuable if most of your code is in Java.

Make sense. Haven’t thought Fabric would have those dependencies… Though seeing Answers stats it is probably unavoidable.

Hi Mateusz!

Great guide, thank you. It is good to know that someone else figured out how to use binding generation despite the somewhat “limited” documentation.

I get the hint about the freezing IDE. In the next release (beta3 or final 1.3) we will also have a Gradle task to run the generation.

Best Regards,
Gergely

We have a few options here:

  • MOE 2.0 will have native stack traces that include mangled, but easily decipherable Java class / method names like Class_methodname because of the new LLVM backend. We really just need to clean the code up a bit to release an alpha.

  • If there would be a crash reporting tool that had an open-source Android SDK, it could be ported to MOE relatively painlessly. HockeyApp’s SDK is open source for example. The main entry point for all such crash reporters is the standard Java UncaughtExceptionHandler that is also present in MOE.

  • I don’t know how much Fabric is using from Android. The screenshot only shows a handful of classes. If one of you would create a Hello World APK with only the Fabric / Crashlytics SDK initialized and uploaded it to the Migeran Analyzer, we would have a detailed report on what exactly is missing from MOE to get it running. Then the missing functionality could be stubbed out / reimplemented on top of the iOS APIs.

Hey guys, just wanted to drop and note that I have been using Fabric / Crashlytics successfully since posting the above. The reports I’m getting out of it are very helpful for my purposes. As my game is itself in beta now, I have not tried to update to newer versions of MOE, in an effort to maintain stability as we approach release.

The monkey wrench we should all keep our eyes on though is the fact that apparently Google has acquired Fabric, and we can expect Fabric and Firebase to merge in the near future. Details here:

https://fabric.io/blog/fabric-joins-google?utm_campaign=fabric-marketing&utm_medium=natural

Name: <YOUR_POD_NAME>
Header path: xcode/Pods/<YOUR_POD_NAME>/<YOUR_POD_NAME>
(or another path to pod headers)
Base package name: com.yourapp.ios.pods.<YOUR_POD_NAME>
Import headers:
#import "<YOUR_POD_HEADER_FILE>.h"
(same as the objc import added in native pod usage)

On step 15) , do we have to create .nbc file manually like that ? Is there a tool to scan through Pods/ folder and create the .nbc for us ?

For now you have to do it manually. .nbc files has been introduced quite recently and I don’t really think that this kind of tool would be available.

Wow, LLVM based, this does not sound like “Android AOT port”.
Any chance to see in action ?
Can you run my Collection Performance on it ?
(I can make a dedicated project for that if you can)

Hi Kirill,

sure, please share the project, and we will run the test.

It is actually still based on Android ART, but we developed a new backend for ART’s optimizing compiler, so it does not output machine code with its custom assembler, but generates LLVM IR / bitcode.

Best Regards,
Gergely