Problem generating MOE bindings for Google ML Kit

ekuleshov · December 15, 2020, 5:01pm

I’m having troubles with generating MOE bindings for Google ML Kit TextRecognition framework and would appreciate any help or hints.

The moeNatJGen task gives a bunch of warnings like attributes may not be specified on a category (log is below) and generated MLKTextRecognizer class doesn’t have methods that I see in the MLKTextRecognizer.h header file, e.g.:

- (void)processImage:(MLKVisionImage *)image
          completion:(MLKTextRecognitionCallback)completion NS_SWIFT_NAME(process(_:completion:));

The MLKVisionImage class and other core classes are also not generated.

I pushed a test project and my *.nbc file to github at https://github.com/ekuleshov/mlkit-test

EK

:moeNatJGen
2020-12-15 11:45:13:899 -0500 [main] DEBUG org.moe.natjgen.Main - NatJGen started
2020-12-15 11:45:14:165 -0500 [main] DEBUG org.moe.natjgen.Indexer - Clang options: [-arch, arm64, -x, objective-c, -fmessage-length=133, -std=c99, -fobjc-arc, -fpascal-strings, -ferror-limit=9999999, -DNS_BLOCK_ASSERTIONS=1, -DOBJC_OLD_DISPATCH_PROTOTYPES=0, -isysroot, /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk, -miphoneos-version-min=9.0, -I/Users/eu/Downloads/mlkit-test/xcode/Pods/MLKitTextRecognition/Frameworks/MLKitTextRecognition.framework/Headers, -I/Users/eu/Downloads/mlkit-test/xcode/Pods/MLKitVision/Frameworks/MLKitVision.framework/Headers, -I/Users/eu/Downloads/mlkit-test/xcode/Pods/MLKitTextRecognition/Frameworks/MLKitTextRecognition.framework/Headers, -I/Users/eu/Downloads/mlkit-test/xcode/Pods/MLKitCommon/Frameworks/MLKitCommon.framework/Headers, -I/var/folders/6x/xg4xswbx3nldfk2vdn_ts6mm0000gn/T/NatJGen801312725765969176, -iquote/Users/eu/Downloads/mlkit-test/xcode/Pods/MLKitTextRecognition/Frameworks/MLKitTextRecognition.framework/Headers]
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/System/Library/Frameworks/Foundation.framework/Headers/NSArray.h:109:12: error: attributes may not be specified on a category
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/System/Library/Frameworks/Foundation.framework/Headers/NSArray.h:196:12: error: attributes may not be specified on a category
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/System/Library/Frameworks/Foundation.framework/Headers/NSOrderedSet.h:112:12: error: attributes may not be specified on a category
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/System/Library/Frameworks/Foundation.framework/Headers/NSOrderedSet.h:187:12: error: attributes may not be specified on a category
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/System/Library/Frameworks/OpenGLES.framework/Headers/EAGLDrawable.h:52:12: error: attributes may not be specified on a category
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/System/Library/Frameworks/UIKit.framework/Headers/UIResponder+UIActivityItemsConfiguration.h:16:12: error: attributes may not be specified on a category
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/System/Library/Frameworks/UIKit.framework/Headers/UISceneActivationConditions.h:31:25: error: attributes may not be specified on a category
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/System/Library/Frameworks/UIKit.framework/Headers/UIScreenshotService.h:35:50: error: attributes may not be specified on a category
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/System/Library/Frameworks/UIKit.framework/Headers/NSUserActivity+NSItemProvider.h:12:81: error: attributes may not be specified on a category
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/System/Library/Frameworks/UIKit.framework/Headers/UNNotificationResponse+UIKitAdditions.h:16:25: error: attributes may not be specified on a category
2020-12-15 11:45:14:596 -0500 [main] DEBUG org.moe.natjgen.Indexer - Indexing translation unit
2020-12-15 11:45:16:849 -0500 [main] DEBUG org.moe.natjgen.Indexer - Clang options: [-arch, armv7, -x, objective-c, -fmessage-length=133, -std=c99, -fobjc-arc, -fpascal-strings, -ferror-limit=9999999, -DNS_BLOCK_ASSERTIONS=1, -DOBJC_OLD_DISPATCH_PROTOTYPES=0, -isysroot, /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk, -miphoneos-version-min=9.0, -I/Users/eu/Downloads/mlkit-test/xcode/Pods/MLKitTextRecognition/Frameworks/MLKitTextRecognition.framework/Headers, -I/Users/eu/Downloads/mlkit-test/xcode/Pods/MLKitVision/Frameworks/MLKitVision.framework/Headers, -I/Users/eu/Downloads/mlkit-test/xcode/Pods/MLKitTextRecognition/Frameworks/MLKitTextRecognition.framework/Headers, -I/Users/eu/Downloads/mlkit-test/xcode/Pods/MLKitCommon/Frameworks/MLKitCommon.framework/Headers, -I/var/folders/6x/xg4xswbx3nldfk2vdn_ts6mm0000gn/T/NatJGen801312725765969176, -iquote/Users/eu/Downloads/mlkit-test/xcode/Pods/MLKitTextRecognition/Frameworks/MLKitTextRecognition.framework/Headers]
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/System/Library/Frameworks/Foundation.framework/Headers/NSArray.h:109:12: error: attributes may not be specified on a category
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/System/Library/Frameworks/Foundation.framework/Headers/NSArray.h:196:12: error: attributes may not be specified on a category
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/System/Library/Frameworks/Foundation.framework/Headers/NSOrderedSet.h:112:12: error: attributes may not be specified on a category
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/System/Library/Frameworks/Foundation.framework/Headers/NSOrderedSet.h:187:12: error: attributes may not be specified on a category
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/System/Library/Frameworks/OpenGLES.framework/Headers/EAGLDrawable.h:52:12: error: attributes may not be specified on a category
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/System/Library/Frameworks/UIKit.framework/Headers/UIResponder+UIActivityItemsConfiguration.h:16:12: error: attributes may not be specified on a category
2020-12-15 11:45:17:244 -0500 [main] DEBUG org.moe.natjgen.Indexer - Indexing translation unit
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/System/Library/Frameworks/UIKit.framework/Headers/UISceneActivationConditions.h:31:25: error: attributes may not be specified on a category
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/System/Library/Frameworks/UIKit.framework/Headers/UIScreenshotService.h:35:50: error: attributes may not be specified on a category
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/System/Library/Frameworks/UIKit.framework/Headers/NSUserActivity+NSItemProvider.h:12:81: error: attributes may not be specified on a category
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/System/Library/Frameworks/UIKit.framework/Headers/UNNotificationResponse+UIKitAdditions.h:16:25: error: attributes may not be specified on a category
2020-12-15 11:45:19:706 -0500 [main] DEBUG org.moe.natjgen.CStructManager - c:@S@__darwin_arm_neon_state64: Failed to merge secondary field types
2020-12-15 11:45:19:706 -0500 [main] DEBUG org.moe.natjgen.CStructManager - c:@S@arm_legacy_debug_state: Failed to merge secondary field types
2020-12-15 11:45:19:707 -0500 [main] DEBUG org.moe.natjgen.CStructManager - c:@S@_CALayerIvars: Failed to merge secondary field types
2020-12-15 11:45:19:707 -0500 [main] DEBUG org.moe.natjgen.CStructManager - c:@S@__darwin_arm_neon_state: Failed to merge secondary field types
2020-12-15 11:45:19:710 -0500 [main] DEBUG org.moe.natjgen.Indexer - Generating classes
2020-12-15 11:45:20:083 -0500 [main] INFO org.moe.natjgen.UnitEditContext - Getting compilation unit from /Users/eu/Downloads/mlkit-test/src/main/java/com/google/mlkit/mlkittextrecognition/MLKTextRecognizedLanguage.java
2020-12-15 11:45:20:454 -0500 [main] INFO org.moe.natjgen.UnitEditContext - Getting compilation unit from /Users/eu/Downloads/mlkit-test/src/main/java/com/google/mlkit/mlkittextrecognition/MLKText.java
2020-12-15 11:45:20:508 -0500 [main] INFO org.moe.natjgen.UnitEditContext - Getting compilation unit from /Users/eu/Downloads/mlkit-test/src/main/java/com/google/mlkit/mlkittextrecognition/MLKTextElement.java
2020-12-15 11:45:20:549 -0500 [main] INFO org.moe.natjgen.UnitEditContext - Getting compilation unit from /Users/eu/Downloads/mlkit-test/src/main/java/com/google/mlkit/mlkittextrecognition/MLKTextRecognizer.java
2020-12-15 11:45:20:590 -0500 [main] INFO org.moe.natjgen.UnitEditContext - Getting compilation unit from /Users/eu/Downloads/mlkit-test/src/main/java/com/google/mlkit/mlkittextrecognition/MLKTextLine.java
2020-12-15 11:45:20:624 -0500 [main] INFO org.moe.natjgen.UnitEditContext - Getting compilation unit from /Users/eu/Downloads/mlkit-test/src/main/java/com/google/mlkit/mlkittextrecognition/MLKTextBlock.java
2020-12-15 11:45:20:661 -0500 [main] DEBUG org.moe.natjgen.Main - NatJGen ended in 6.762858196s

Noisyfox · December 17, 2020, 3:24am

All you need to do is change the Header path because this field works as a filter that only classes in the headers that under this path will be generated.

So if you set the path to MLKTextRecognizer’s framework path, then bindings for classes like MLKVisionImage won’t be generated, which then leads to the disappearing of function - (void)processImage:(MLKVisionImage *)image completion:(MLKTextRecognitionCallback)completion NS_SWIFT_NAME(process(_:completion:)); because the dependent class MLKVisionImage is missing.

Screen Shot 2020-12-17 at 2.20.09 pm

ekuleshov · December 17, 2020, 5:31am

@Noisyfox wow! That simple, eh?

I’ve got the bindings generated, but now getting a hard crash inside MLKTextRecognizer.process(..) call while running in a simulator.

Any tips how to debug that or what could be wrong? I pushed updated code to the same github repo.

Noisyfox · December 22, 2020, 12:18am

The issue in your your code is this line:

MLKTextRecognizer recognizer = MLKTextRecognizer.alloc().init();

If you look at the header file MLKTextRecognizer.h, you will see:

/**
 * Returns a text recognizer.
 *
 * @return A text recognizer.
 */
+ (instancetype)textRecognizer NS_SWIFT_NAME(textRecognizer());

/** Unavailable. Use the class method. */
- (instancetype)init NS_UNAVAILABLE;

And in Google’s doc there is:

MLKTextRecognizer *textRecognizer = [MLKTextRecognizer textRecognizer];

So you should simply use

MLKTextRecognizer recognizer = MLKTextRecognizer.textRecognizer();

in your code instead.

Also you should not use dispatch_sync since the block callback of processImageCompletion is already running in main thread.

ekuleshov · December 22, 2020, 12:37am

@Noisyfox I can’t believe I missed that. You are my saviour again! Thank you a lot.

ekuleshov · December 22, 2020, 12:42am

BTW, when those bindings are generated, what would it take to pull the docs comments from the *.h files and add them into the generated Java classes?

Noisyfox · December 22, 2020, 12:49am

That sounds like an interesting idea. I’ll have a look.

ekuleshov · December 26, 2020, 6:53pm

If that is mostly text processing I can look at that too. Can you give me some pointers to the code?

Noisyfox · December 30, 2020, 6:47am

It’s actually not that straightforward, it involves getting comments using libclang when indexing all the classes/methods/etc.

I managed to make this working, if you are interested you could find the changes here

This also allows generating iOS SDK bindings with comments, which will be added in the next release.

However, since the init method in MLKTextRecognizer.h is marked as NS_UNAVAILABLE, the comment of this method will be ignored. There might be way of keeping it however it requires some major changes so I decided not doing that atm.

ekuleshov · December 30, 2020, 4:55pm

@Noisyfox this is pretty cool! I was silently hoping the comments will make into the iOS SDK bindings. Thank you.

So, when are you planning for the next release?

BTW, speaking of the bindings. Unless I’m missing something, in a multi-module project, the generate bindings action in the bindings editor always kick in building of the all bindings (in all modules from the project root), instead of only ones declared in the current .nbc file or in current module.

Noisyfox · December 31, 2020, 5:01am

It should be out in a few days.

That doesn’t sound right to me. It should only generate bindings declared in the current .nbc file.

ekuleshov · December 31, 2020, 4:52pm

I was looking at the log output in the “MOE” tool window in AS. I can try to create some test project if it would help.

BTW, how does binding generator decide on the paths for the .h files and all? Is it relative to the project root, to the module root, xcode folder or relative to the .nbc file?

PS: eagerly waiting for the next release with the updated bindings. Thank you for your work

Noisyfox · January 1, 2021, 1:04am

Yes that would be helpful.

If you take a look at the output log, you could find lines start with “Clang options” like

Clang options: [-arch, armv7, -x, objective-c, -fmessage-length=133, -std=c99, -fobjc-arc, -fpascal-strings, -ferror-limit=9999999, -DNS_BLOCK_ASSERTIONS=1, -DOBJC_OLD_DISPATCH_PROTOTYPES=0, -isysroot, /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk, -miphoneos-version-min=9.0, -I/Users/noisyfox/Downloads/mlkit-test-master/xcode/Pods, -I/Users/noisyfox/Downloads/mlkit-test-master/xcode/Pods/MLKitVision/Frameworks/MLKitVision.framework/Headers, -I/Users/noisyfox/Downloads/mlkit-test-master/xcode/Pods/MLKitTextRecognition/Frameworks/MLKitTextRecognition.framework/Headers, -I/Users/noisyfox/Downloads/mlkit-test-master/xcode/Pods/MLKitCommon/Frameworks/MLKitCommon.framework/Headers, -I/var/folders/gh/w_h85ktx7gdbw3s0wbgwyt4r0000gn/T/NatJGen5454023824443761952, -iquote/Users/noisyfox/Downloads/mlkit-test-master/xcode/Pods, -fparse-all-comments]

And you could find several paths start with -I or -iquote, which are the header search paths.
There might be other paths appears in the options, the meaning of those paths can be found here:

github.com

multi-os-engine-community/moe-natjgen/blob/a7bf84012dc3ba127b9b1614f700ba1f552df16b/src/main/java/org/moe/natjgen/Configuration.java#L1471-L1491


// Add include paths for '#include <...>'
for (String path : getHeaderSearchPaths()) {
    cmdlineArgs.add("-I" + getDemangledPath(path));
}
if (getUseIncludedStandardHeaders()) {
    try {
        cmdlineArgs.add("-I" + StdHeaders.getPath());
    } catch (RuntimeException ex) {
        LOG.error("Failed to add standard header search path", ex);
    }
}

// Add include paths for '#include "..."'
for (String path : getUserHeaderSearchPaths()) {
    cmdlineArgs.add("-iquote" + getDemangledPath(path));
}

// Add include paths for '#include <.../...>'
for (String path : getFrameworkSearchPaths()) {
    cmdlineArgs.add("-F" + getDemangledPath(path));
}