Google 的 ML Kit Flutter 插件

Pub Version

一个用于 Android 和 iOS 的 Flutter 插件，可使用 Google 的独立 ML Kit。

功能

视觉

功能 Android iOS
文本识别 ✅ ✅
面部检测 ✅ ✅
姿态检测 ✅ ✅
自拍分割 尚不确定 尚不确定
条形码扫描 ✅ ✅
图像标记 ✅ ✅
物体检测与跟踪 ✅ 尚不确定
数字墨水识别 ✅ ✅
文本检测器 V2 ✅ 尚不确定

自然语言

功能 Android iOS
语言识别 ✅ ✅
设备端翻译 ✅ 尚不确定
智能回复 ✅ 尚不确定
实体提取 ✅ 尚不确定

要求

iOS

  • 最低 iOS 部署目标：10.0
  • Xcode 12 或更高版本
  • Swift 5
  • ML Kit 仅支持 64 位架构 (x86_64 和 arm64)。请查看此 列表 以了解您的设备是否具有所需的设备功能。

由于 ML Kit 不支持 32 位架构 (i386 和 armv7) (阅读更多)，因此在运行 flutter build ios 或 flutter build ipa 时，您需要在 Xcode 中排除 armv7 架构。

前往 项目 > Runner > Build Settings > Excluded Architectures > Any SDK > armv7

然后您的 Podfile 应如下所示：

# add this line:
$iOSVersion = '10.0'

post_install do |installer|
  # add these lines:
  installer.pods_project.build_configurations.each do |config|
    config.build_settings["EXCLUDED_ARCHS[sdk=*]"] = "armv7"
    config.build_settings['IPHONEOS_DEPLOYMENT_TARGET'] = $iOSVersion
  end
  
  installer.pods_project.targets.each do |target|
    flutter_additional_ios_build_settings(target)
    
    # add these lines:
    target.build_configurations.each do |config|
      if Gem::Version.new($iOSVersion) > Gem::Version.new(config.build_settings['IPHONEOS_DEPLOYMENT_TARGET'])
        config.build_settings['IPHONEOS_DEPLOYMENT_TARGET'] = $iOSVersion
      end
    end
    
  end
end

请注意，最低 IPHONEOS_DEPLOYMENT_TARGET 为 10.0，您可以将其设置为更新的版本，但不能旧于此。

Android

  • minSdkVersion: 21
  • targetSdkVersion: 29

用法

将此插件添加为 pubspec.yaml 中的依赖项。

  • 在您的项目级 build.gradle 文件中，确保在 buildscript 和 allprojects 部分都包含 Google 的 Maven 存储库（适用于所有 API）。

  • 除 图像标记、面部检测 和 条形码扫描 之外的所有 API 都使用捆绑模型，因此其他 API 应该开箱即用。

  • 对于使用非捆绑模型的 API，通过将以下内容添加到您应用的 AndroidManifest.xml 来配置您的应用程序，使其可以从 Play 商店自动将模型下载到您的设备。如果未配置，则在首次调用相应 API 时将下载模型。

    <meta-data
        android:name="com.google.mlkit.vision.DEPENDENCIES"
        android:value="ica" />
    <!-- To use multiple models: android:value="ica,model2,model3" -->

    使用这些选项：

    • ica – 图像标记
    • ocr – 条形码扫描
    • face – 面部检测

1. 创建 InputImage

从路径

final inputImage = InputImage.fromFilePath(filePath);

从文件

final inputImage = InputImage.fromFile(file);

从字节

final inputImage = InputImage.fromBytes(bytes: bytes, inputImageData: inputImageData);

从 CameraImage（如果您正在使用 camera 插件）

final camera; // your camera instance
final WriteBuffer allBytes = WriteBuffer();
for (Plane plane in cameraImage.planes) {
  allBytes.putUint8List(plane.bytes);
}
final bytes = allBytes.done().buffer.asUint8List();

final Size imageSize = Size(cameraImage.width.toDouble(), cameraImage.height.toDouble());

final InputImageRotation imageRotation =
    InputImageRotationMethods.fromRawValue(camera.sensorOrientation) ??
        InputImageRotation.Rotation_0deg;

final InputImageFormat inputImageFormat =
    InputImageFormatMethods.fromRawValue(cameraImage.format.raw) ??
        InputImageFormat.NV21;

final planeData = cameraImage.planes.map(
  (Plane plane) {
    return InputImagePlaneMetadata(
      bytesPerRow: plane.bytesPerRow,
      height: plane.height,
      width: plane.width,
    );
  },
).toList();

final inputImageData = InputImageData(
  size: imageSize,
  imageRotation: imageRotation,
  inputImageFormat: inputImageFormat,
  planeData: planeData,
);

final inputImage = InputImage.fromBytes(bytes: bytes, inputImageData: inputImageData);

2. 创建检测器实例

// vision
final barcodeScanner = GoogleMlKit.vision.barcodeScanner();
final digitalInkRecogniser = GoogleMlKit.vision.digitalInkRecogniser();
final faceDetector = GoogleMlKit.vision.faceDetector();
final imageLabeler = GoogleMlKit.vision.imageLabeler();
final poseDetector = GoogleMlKit.vision.poseDetector();
final textDetector = GoogleMlKit.vision.textDetector();
final objectDetector = GoogleMlKit.vision.objectDetector(CustomObjectDetectorOptions or ObjectDetectorOptions);

// nl
final entityExtractor = GoogleMlKit.nlp.entityExtractor();
final languageIdentifier = GoogleMlKit.nlp.languageIdentifier();
final onDeviceTranslator = GoogleMlKit.nlp.onDeviceTranslator();
final smartReply = GoogleMlKit.nlp.smartReply();

// managing models
final translateLanguageModelManager = GoogleMlKit.nlp.translateLanguageModelManager();
final entityModelManager = GoogleMlKit.nlp.entityModelManager();
final remoteModelManager = GoogleMlKit.vision.remoteModelManager();

3. 调用相应的方法

// vision
final List<Barcode> barcodes = await barcodeScanner.processImage(inputImage);
final List<RecognitionCandidate> canditates = await digitalInkRecogniser.readText(points, languageTag);
final List<Face> faces = await faceDetector.processImage(inputImage);
final List<ImageLabel> labels = await imageLabeler.processImage(inputImage);
final List<Pose> poses = await poseDetector.processImage(inputImage);
final RecognisedText recognisedText = await textDetector.processImage(inputImage);
final List<DetectedObject> objects = await objectDetector.processImage(inputImage);

// nl
final List<EntityAnnotation> entities = await entityExtractor.extractEntities(text, filters, locale, timezone);
final bool response = await entityModelManager.downloadModel(modelTag);
final String response = await entityModelManager.isModelDownloaded(modelTag);
final String response = await entityModelManager.deleteModel(modelTag);
final List<String> availableModels = await entityModelManager.getAvailableModels();
try {
  final String response = await languageIdentifier.identifyLanguage(text);
} on PlatformException catch (pe) {
  if (pe.code == languageIdentifier.errorCodeNoLanguageIdentified) {
    // no language detected
  }
  // other plugin error
}
try {
  final List<IdentifiedLanguage> response = await languageIdentifier.identifyPossibleLanguages(text);
} on PlatformException catch (pe) {
  if (pe.code == languageIdentifier.errorCodeNoLanguageIdentified) {
    // no language detected
  }
  // other plugin error
}
final String response = await onDeviceTranslator.translateText(text);
final bool response = await translateLanguageModelManager.downloadModel(modelTag);
final String response = await translateLanguageModelManager.isModelDownloaded(modelTag);
final String response = await translateLanguageModelManager.deleteModel(modelTag);
final List<String> availableModels = await translateLanguageModelManager.getAvailableModels();
final List<SmartReplySuggestion> suggestions = await smartReply.suggestReplies();
// add conversations for suggestions
smartReply.addConversationForLocalUser(text);
smartReply.addConversationForRemoteUser(text, userID);

4. 从响应中提取数据。

a. 提取条形码。

for (Barcode barcode in barcodes) {
  final BarcodeType type = barcode.type;
  final Rect boundingBox = barcode.value.boundingBox;
  final String displayValue = barcode.value.displayValue;
  final String rawValue = barcode.value.rawValue;

  // See API reference for complete list of supported types
  switch (type) {
    case BarcodeType.wifi:
      BarcodeWifi barcodeWifi = barcode.value;
      break;
    case BarcodeValueType.url:
      BarcodeUrl barcodeUrl = barcode.value;
      break;
  }
}

b. 提取面部。

for (Face face in faces) {
  final Rect boundingBox = face.boundingBox;

  final double rotY = face.headEulerAngleY; // Head is rotated to the right rotY degrees
  final double rotZ = face.headEulerAngleZ; // Head is tilted sideways rotZ degrees

  // If landmark detection was enabled with FaceDetectorOptions (mouth, ears,
  // eyes, cheeks, and nose available):
  final FaceLandmark leftEar = face.getLandmark(FaceLandmarkType.leftEar);
  if (leftEar != null) {
    final Point<double> leftEarPos = leftEar.position;
  }

  // If classification was enabled with FaceDetectorOptions:
  if (face.smilingProbability != null) {
    final double smileProb = face.smilingProbability;
  }

  // If face tracking was enabled with FaceDetectorOptions:
  if (face.trackingId != null) {
    final int id = face.trackingId;
  }
}

c. 提取标签。

for (ImageLabel label in labels) {
  final String text = label.text;
  final int index = label.index;
  final double confidence = label.confidence;
}

d. 提取文本。

String text = recognisedText.text;
for (TextBlock block in recognisedText.blocks) {
  final Rect rect = block.rect;
  final List<Offset> cornerPoints = block.cornerPoints;
  final String text = block.text;
  final List<String> languages = block.recognizedLanguages;

  for (TextLine line in block.lines) {
    // Same getters as TextBlock
    for (TextElement element in line.elements) {
      // Same getters as TextBlock
    }
  }
}

e. 姿态检测

for (Pose pose in poses) {
  // to access all landmarks
  pose.landmarks.forEach((_, landmark) {
    final type = landmark.type;
    final x = landmark.x;
    final y = landmark.y;
  }
  
  // to access specific landmarks
  final landmark = pose.landmarks[PoseLandmarkType.nose];
}

f. 数字墨水识别

for (final candidate in candidates) {
  final text = candidate.text;
  final score = candidate.score;
}

g. 提取建议

//status implications
//1 = Language Not Supported
//2 = Can't determine a reply
//3 = Successfully generated 1-3 replies
int status = result['status'];

List<SmartReplySuggestion> suggestions = result['suggestions'];

h. 提取对象

for(DetectedObject detectedObject in _objects){
  final rect = detectedObject.getBoundinBox();
  final trackingId = detectedObject.getTrackingId();

  for(Label label in detectedObject.getLabels()){
    print('${label.getText()} ${label.getConfidence()}');
  }
}

5. 使用 close() 释放资源。

// vision
barcodeScanner.close();
digitalInkRecogniser.close();
faceDetector.close();
imageLabeler.close();
poseDetector.close();
textDetector.close();
objectDetector.close();

// nl
entityExtractor.close();
languageIdentifier.close();
onDeviceTranslator.close();
smartReply.close();

示例应用

请查看此 示例，了解该插件的实际应用。

从 ML Kit for Firebase 迁移

从 ML Kit for Firebase 迁移时，请阅读 此指南。有关 Android 的详细信息，请阅读 此。有关 iOS 的详细信息，请阅读 此。

已知问题

Android

要减小 APK 大小，请在问题 #26 中了解更多信息。另请参阅 此。

iOS

如果您在应用中使用了此插件以及任何其他需要 Firebase 的插件，在运行 pod install 时会遇到依赖关系错误。要了解更多信息，请查看问题 #27。

贡献

欢迎贡献。
如有任何问题，请提交 issue。
对于非微小的修复，请在提交 pull request 之前先创建一个 issue。
对于微小的修复，请直接提交 pull request。

许可证

MIT

GitHub

https://github.com/bharat-biradar/Google-Ml-Kit-plugin