twilio_programmable_video

用于 Twilio Programmable Video 的 Flutter 插件,它使您能够构建实时视频通话应用程序(WebRTC)。此 Flutter 插件是社区维护的项目,由 Twilio 维护。如果您有任何问题,请提交问题,而不是联系支持。

此包目前正在开发中,不应用于生产应用程序。在我们达到 v1.0.0 之前,我们不能保证当前 API 实现会在版本之间保持不变。

示例

请查看我们在此插件中提供的详尽的 示例

Twilio Programmable Video Example

加入社区

如果您有任何问题或困难,请加入我们的 Discord

常见问题

在创建新问题之前,请先阅读 常见问题解答

支持的平台

  • Android
  • iOS
  • Web(查看预发布版本)

入门

先决条件

在开始使用该插件之前,您需要确保您的项目已设置好所有内容。

Android

为了让此插件在 Android 上正常运行,您需要调整一些文件。

权限

打开您 android/app/src/main 目录中的 AndroidManifest.xml 文件,并添加以下设备权限

...
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.CAMERA"/>
<uses-permission android:name="android.permission.BLUETOOTH"/>
<uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS"/>
    ...
Proguard

将以下行添加到您的 android/app/proguard-rules.pro 文件中。

-keep class tvi.webrtc.** { *; }
-keep class com.twilio.video.** { *; }
-keep class com.twilio.common.** { *; }
-keepattributes InnerClasses

别忘了在您的 android/app/build.gradle 文件中引用此 proguard-rules.pro

android {

    ...

    buildTypes {

        release {

            ...

            minifyEnabled true
            useProguard true

            proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'

        }
    }
}

iOS

为了让此插件在 iOS 上正常运行,您需要调整一些文件。

权限

打开您 ios/Runner 目录中的 Info.plist 文件,并添加以下权限

...
<key>NSCameraUsageDescription</key>
<string>Your message to user when the camera is accessed for the first time</string>
<key>NSMicrophoneUsageDescription</key>
<string>Your message to user when the microphone is accessed for the first time</string>
<key>io.flutter.embedded_views_preview</key>
<true/>
...

打开您 ios 目录中的 Podfile 文件,并添加以下权限

...
post_install do |installer|
  installer.pods_project.targets.each do |target|
    flutter_additional_ios_build_settings(target)
    target.build_configurations.each do |config|
      config.build_settings['GCC_PREPROCESSOR_DEFINITIONS'] ||= [
        '$(inherited)',
        'PERMISSION_CAMERA=1',
        'PERMISSION_MICROPHONE=1',
        # Add other permissions required by your app
      ]

    end
  end
end
...

有关示例,请查看示例应用程序的 Podfile

将最低 iOS 版本设置为 11
  1. 在 Xcode 中,打开您应用程序 ios 文件夹中的 Runner.xcworkspace
  2. 要查看您应用程序的设置,请在 Xcode 项目导航器中选择 Runner 项目。然后,在主视图侧边栏中,选择 Runner 目标。
  3. 选择 General 选项卡。
  4. Deployment Info 部分,将 Target 设置为 iOS 11。
后台模式

要允许在应用程序后台运行时保持与房间的连接,您必须从 Capabilities 项目设置页面中选择 Audio、AirPlay 和 Picture in Picture 后台模式。有关更多信息,请参阅 Twilio 文档

连接到房间

在您的 Flutter 应用程序中,调用 TwilioProgrammableVideo.connect() 来连接到房间。连接后,您可以与其他连接到房间的参与者进行音视频流的发送和接收。

Room _room;
final Completer<Room> _completer = Completer<Room>();

void _onConnected(Room room) {
  print('Connected to ${room.name}');
  _completer.complete(_room);
}

void _onConnectFailure(RoomConnectFailureEvent event) {
  print('Failed to connect to room ${event.room.name} with exception: ${event.exception}');
  _completer.completeError(event.exception);
}

Future<Room> connectToRoom() async {
  // Retrieve the camera source of your choosing
  var cameraSources = await CameraSource.getSources();
  var cameraCapturer = CameraCapturer(
    cameraSources.firstWhere((source) => source.isFrontFacing),
  );

  var connectOptions = ConnectOptions(
    accessToken,
    roomName: roomName,                   // Optional name for the room
    region: region,                       // Optional region.
    preferredAudioCodecs: [OpusCodec()],  // Optional list of preferred AudioCodecs
    preferredVideoCodecs: [H264Codec()],  // Optional list of preferred VideoCodecs.
    audioTracks: [LocalAudioTrack(true)], // Optional list of audio tracks.
    dataTracks: [
      LocalDataTrack(
        DataTrackOptions(
          ordered: ordered,                      // Optional, Ordered transmission of messages. Default is `true`.
          maxPacketLifeTime: maxPacketLifeTime,  // Optional, Maximum retransmit time in milliseconds. Default is [DataTrackOptions.defaultMaxPacketLifeTime]
          maxRetransmits: maxRetransmits,        // Optional, Maximum number of retransmitted messages. Default is [DataTrackOptions.defaultMaxRetransmits]
          name: name                             // Optional
        ),                                // Optional
      ),
    ],                                    // Optional list of data tracks
    videoTracks: [LocalVideoTrack(true, cameraCapturer)], // Optional list of video tracks.
  );
  _room = await TwilioProgrammableVideo.connect(connectOptions);
  _room.onConnected.listen(_onConnected);
  _room.onConnectFailure.listen(_onConnectFailure);
  return _completer.future;
}

连接到房间时,您必须提供 访问令牌

加入房间

如果您想加入一个已知存在的房间,您可以以与创建房间完全相同的方式处理:只需将房间名称传递给 connect 方法即可。进入房间后,您将为每个成功加入的参与者收到一个 RoomParticipantConnectedEvent。查询 room.remoteParticipants getter 将返回已加入房间的任何现有参与者。

Room _room;
final Completer<Room> _completer = Completer<Room>();

void _onConnected(Room room) {
  print('Connected to ${room.name}');
  _completer.complete(_room);
}

void _onConnectFailure(RoomConnectFailureEvent event) {
  print('Failed to connect to room ${event.room.name} with exception: ${event.exception}');
  _completer.completeError(event.exception);
}

Future<Room> connectToRoom() async {
  // Retrieve the camera source of your choosing
  var cameraSources = await CameraSource.getSources();
  var cameraCapturer = CameraCapturer(
    cameraSources.firstWhere((source) => source.isFrontFacing),
  );

  var connectOptions = ConnectOptions(
    accessToken,
    roomName: roomName,
    region: region,                       // Optional region.
    preferAudioCodecs: [OpusCodec()],     // Optional list of preferred AudioCodecs
    preferVideoCodecs: [H264Codec()],     // Optional list of preferred VideoCodecs.
    audioTracks: [LocalAudioTrack(true)], // Optional list of audio tracks.
    dataTracks: [
      LocalDataTrack(
        DataTrackOptions(
          ordered: ordered,                      // Optional, Ordered transmission of messages. Default is `true`.
          maxPacketLifeTime: maxPacketLifeTime,  // Optional, Maximum retransmit time in milliseconds. Default is [DataTrackOptions.defaultMaxPacketLifeTime]
          maxRetransmits: maxRetransmits,        // Optional, Maximum number of retransmitted messages. Default is [DataTrackOptions.defaultMaxRetransmits]
          name: name                             // Optional
        ),                                // Optional
      ),
    ],                                    // Optional list of data tracks
    videoTracks([LocalVideoTrack(true, cameraCapturer)]), // Optional list of video tracks.
  );
  _room = await TwilioProgrammableVideo.connect(connectOptions);
  _room.onConnected.listen(_onConnected);
  _room.onConnectFailure.listen(_onConnectFailure);
  return _completer.future;
}

设置本地媒体

您可以通过以下方式从设备的麦克风或摄像头捕获本地媒体:

// Create an audio track.
var localAudioTrack = LocalAudioTrack(true);

// Retrieve the camera source of your choosing
var cameraSources = await CameraSource.getSources();
var cameraCapturer = CameraCapturer(
  cameraSources.firstWhere((source) => source.isFrontFacing),
);

// Create a video track.
var localVideoTrack = LocalVideoTrack(true, cameraCapturer);

// Getting the local video track widget.
// This can only be called after the TwilioProgrammableVideo.connect() is called.
var widget = localVideoTrack.widget();

连接为仅发布参与者

目前无法连接为仅发布参与者。

处理远程参与者

处理已连接的参与者

当您加入房间时,可能会有参与者已经存在。您可以在调用 Room.onConnected 侦听器时,通过使用 room.remoteParticipants getter 来检查现有参与者。

// Connect to a room.
var room = await TwilioProgrammableVideo.connect(connectOptions);

room.onConnected((Room room) {
  print('Connected to ${room.name}');
});

room.onConnectFailure((RoomConnectFailureEvent event) {
    print('Failed connecting, exception: ${event.exception.message}');
});

room.onDisconnected((RoomDisconnectEvent event) {
  print('Disconnected from ${event.room.name}');
});

room.onRecordingStarted((Room room) {
  print('Recording started in ${room.name}');
});

room.onRecordingStopped((Room room) {
  print('Recording stopped in ${room.name}');
});

// ... Assume we have received the connected callback.

// After receiving the connected callback the LocalParticipant becomes available.
var localParticipant = room.localParticipant;
print('LocalParticipant ${room.localParticipant.identity}');

// Get the first participant from the room.
var remoteParticipant = room.remoteParticipants[0];
print('RemoteParticipant ${remoteParticipant.identity} is in the room');

处理参与者连接事件

当参与者连接或断开您正在连接的房间时,您将通过事件侦听器收到通知。这些事件可帮助您的应用程序跟踪加入或离开房间的参与者。

// Connect to a room.
var room = await TwilioProgrammableVideo.connect(connectOptions);

room.onParticipantConnected((RoomParticipantConnectedEvent event) {
  print('Participant ${event.remoteParticipant.identity} has joined the room');
});

room.onParticipantDisconnected((RoomParticipantDisconnectedEvent event) {
  print('Participant ${event.remoteParticipant.identity} has left the room');
});

显示远程参与者的 Widget

要查看远程参与者发送的视频轨道,我们需要将他们的 Widget 添加到树中。

room.onParticipantConnected((RoomParticipantConnectedEvent roomEvent) {
  // We can respond when the Participant adds a VideoTrack by adding the widget to the tree.
  roomEvent.remoteParticipant.onVideoTrackSubscribed((RemoteVideoTrackSubscriptionEvent event) {
    var mirror = false;
    _widgets.add(event.remoteParticipant.widget(mirror));
  });
});

使用 DataTrack API

DataTrack API 允许您创建一个 DataTrack 通道,该通道可用于向订阅了数据的零个或多个接收者发送低延迟消息。

目前,开始使用 DataTrack 的唯一方法是在 连接到房间 时在 ConnectOptions 中指定它。

连接到房间后,您必须等待收到 LocalDataTrackPublishedEvent,然后才能开始向该轨道发送数据。您可以使用 Room.onConnected 侦听器在连接到房间后开始收听此事件。

// Connect to a room.
var room = await TwilioProgrammableVideo.connect(connectOptions);

room.onConnected((Room room) {
  // Once connected to the room start listening for the moment the LocalDataTrack gets published to the room.
  room.localParticipant.onDataTrackPublished.listen(_onLocalDataTrackPublished);
});

  // Once connected to the room start listening for the moment the LocalDataTrack gets published to the room.
  event.room.localParticipant.onDataTrackPublished.listen(_onLocalDataTrackPublished);
});

void _onLocalDataTrackPublished(LocalDataTrackPublishedEvent event) {
  // This event contains a localDataTrack you can use to send data.
  event.localDataTrackPublication.localDataTrack.send('Hello world');
}

如果您想接收来自 RemoteDataTrack 的数据,您必须在 RemoteParticipant 开始发布该轨道并且您已订阅它之后开始收听该轨道。

// Connect to a room.
var room = await TwilioProgrammableVideo.connect(connectOptions);

room.onParticipantConnected((RoomParticipantConnectedEvent event) {
  // A participant connected, now you can start listening to RemoteParticipant events
  event.remoteParticipant.onDataTrackSubscribed.listen(_onDataTrackSubscribed)
});

void _onDataTrackSubscribed(RemoteDataTrackSubscriptionEvent event) {
  final dataTrack = event.remoteDataTrackPublication.remoteDataTrack;
  dataTrack.onMessage.listen(_onMessage);
}

void _onMessage(RemoteDataTrackStringMessageEvent event) {
  print('onMessage => ${event.remoteDataTrack.sid}, ${event.message}');
}

请记住,您将不会收到在您开始收听之前发送的消息。

参与房间

显示摄像头预览

就像 Twilio 一样,我们完全理解您希望在进入房间前看起来很棒。

// Provide a `create` function to the `LocalVideoTrack` class that will trigger initialization at the native layer.
await localVideoTrack?.create(); 

//Add a publishTrack method to LocalParticipants to allow for publishing LocalVideoTracks as needed.
await localVideoTrack?.publish();

断开与房间的连接

您可以断开与您当前参与的房间的连接。其他参与者将收到 RoomParticipantDisconnectedEvent

// To disconnect from a Room, we call:
await room.disconnect();

// This results in a call to Room#onDisconnected
room.onDisconnected((RoomDisconnectEvent event) {
  print('Disconnected from ${event.room.name}');
});

房间重新连接

由于信令或媒体重新连接事件,会触发房间重新连接。

/// Exception will be either TwilioException.signalingConnectionDisconnectedException or TwilioException.mediaConnectionErrorException
room.onReconnecting((RoomReconnectingEvent event) {
  print('Reconnecting to room ${event.room.name}, exception = ${event.exception.message}');
});

room.onReconnected((Room room) {
  print('Reconnected to room ${room.name}');
});

配置音频、视频输入和输出设备

利用控制输入和输出设备的能力,您可以构建更好的最终用户体验。

选择特定的视频输入

CameraCapturer 类用于从给定的 CameraSource 提供 LocalVideoTrack 的视频帧。

// Share your camera.
var cameraSources = await CameraSource.getSources();
var cameraCapturer = CameraCapturer(
  cameraSources.firstWhere((source) => source.isFrontFacing),
);
var localVideoTrack = LocalVideoTrack(true, cameraCapturer);

// Render camera to a widget (only after connect event).
var mirror = true;
var widget = localVideoTrack.widget(mirror);
_widgets.add(widget);

// Switch the camera source.
var cameraSources = await CameraSource.getSources();
var cameraSource = cameraSources.firstWhere((source) => source.isBackFacing);
await cameraCapturer.switchCamera(cameraSource);
await primaryVideoView.setMirror(cameraSource.isBackFacing);

选择特定的音频输出

使用 TwilioProgrammableVideo 类,您可以指定音频是通过耳机、扬声器还是可用的蓝牙音频设备路由。

笔记

如果 speakerphoneEnabledbluetoothPreferred 都为 true,则将在可用时使用蓝牙音频设备,否则音频将通过扬声器路由。

// Route audio through speaker
await TwilioProgrammableVideo.setAudioSettings(speakerphoneEnabled: true, bluetoothPreferred: false);

// Route audio through headset
await TwilioProgrammableVideo.setAudioSettings(speakerphoneEnabled: false, bluetoothPreferred: false);

// Use Bluetooth if available, otherwise use the headset.
await TwilioProgrammableVideo.setAudioSettings(speakerphoneEnabled: false, bluetoothPreferred: true);

// Use Bluetooth if available, otherwise use the speaker.
await TwilioProgrammableVideo.setAudioSettings(speakerphoneEnabled: true, bluetoothPreferred: true);

笔记

调用 setAudioSettings 后,Android 和 iOS 实现将监听路由更改,并努力确保继续使用应用的音频设置。虽然是这样,您可以使用 TwilioProgrammableVideo 类监听此类更改。

TwilioProgrammableVideo.onAudioNotification.listen((event) {
// do things.
});

要禁用音频设置管理和路由更改观察,请使用 TwilioProgrammableVideo 类调用 disableAudioSettings

await TwilioProgrammableVideo.disableAudioSettings();

播放音频文件以提供丰富的用户体验

iOS

为了在该插件中使用时播放音频文件,我们推荐使用 ocarina 插件(v0.1.2 及更高版本)。

此建议是在调查了 Flutter 生态系统中提供此功能的可用插件之后得出的,以找出与此插件配合良好的插件。

观察到的其他提供此功能的插件的主要问题是,在 iOS 上,大多数插件会修改 AVAudioSession 模式,将其置于仅播放模式,因此阻止视频通话录制音频。

iOS 上音频文件播放的第二个问题是,操作系统会优先处理 VoiceProcessingIO Audio Unit,导致在使用此 AudioUnit 时,其他音频源的播放音量大大降低。为了解决这个问题,我们提供了自定义的 AVAudioEngineDevice,插件用户可以启用它,并提供如下示例。AVAudioEngineDevice 是从 Twilio 的示例改编而来的,旨在为从该插件到 AVAudioEngineDevice 的音频文件播放和管理提供一个接口。

要启用 AVAudioEngineDevice 的使用,并将音频文件播放管理从 ocarina 委托给它,请按如下方式更新您的 AppDelegate.swiftdidFinishLaunch 方法。

  override func application(
    _ application: UIApplication,
    didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?
  ) -> Bool {
    GeneratedPluginRegistrant.register(with: self)

    let audioDevice = AVAudioEngineDevice.getInstance()
    SwiftTwilioProgrammableVideoPlugin.setCustomAudioDevice(
        audioDevice,
        onConnected: audioDevice.onConnected,
        onDisconnected: audioDevice.onDisconnected)
    SwiftOcarinaPlugin.useDelegate(
        load: audioDevice.addMusicNode,
        dispose: audioDevice.disposeMusicNode,
        play: audioDevice.playMusic,
        pause: audioDevice.pauseMusic,
        resume: audioDevice.resumeMusic,
        stop: audioDevice.stopMusic,
        volume: audioDevice.setMusicVolume,
        seek: audioDevice.seekPosition,
        position: audioDevice.getPosition
    )

    return super.application(application, didFinishLaunchingWithOptions: launchOptions)
  }

完成此操作后,您应该能够像往常一样继续使用此插件和 ocarina

Android

从版本 0.11.0 开始,我们还在 Android 上提供了与 ocarina 的集成。

此次集成的目的是允许根据播放状态智能管理音频设置和音频焦点。

要获得此集成的优势,请将以下内容添加到您的 MainActivity.kt

    private lateinit var PACKAGE_ID: String

    @RequiresApi(Build.VERSION_CODES.O)
    override fun configureFlutterEngine(@NonNull flutterEngine: FlutterEngine) {
        super.configureFlutterEngine(flutterEngine)
        PACKAGE_ID = applicationContext.packageName
        OcarinaPlugin.addListener(PACKAGE_ID, TwilioProgrammableVideoPlugin.getAudioPlayerEventListener());
    }

    override fun cleanUpFlutterEngine(@NonNull flutterEngine: FlutterEngine) {
        super.cleanUpFlutterEngine(flutterEngine)
        OcarinaPlugin.removeListener(PACKAGE_ID)
    }

启用调试日志记录

使用 TwilioProgrammableVideo 类,您可以启用插件的原生和 Dart 日志记录。

var nativeEnabled = true;
var dartEnabled = true;
TwilioProgrammableVideo.debug(native: nativeEnabled, dart: dartEnabled);

访问令牌

请记住,您无法使用 TestCredentials 为 programmable-video 生成访问令牌,请使用 LIVE 凭据。

您可以在 Twilio 控制台的 测试工具 中轻松生成访问令牌来开始测试您的代码。但我们建议您设置后端来为您生成这些令牌并保护您的 Twilio 凭据。就像我们在 示例应用程序 中所做的那样。

事件表

该插件当前支持的所有事件的参考表

类型 事件流 事件数据 已实现
LocalParticipant onAudioTrackPublished LocalAudioTrackPublishedEvent
LocalParticipant onAudioTrackPublicationFailed LocalAudioTrackPublicationFailedEvent
LocalParticipant onDataTrackPublished LocalDataTrackPublishedEvent
LocalParticipant onDataTrackPublicationFailed LocalDataTrackPublicationFailedEvent
LocalParticipant onVideoTrackPublished LocalVideoTrackPublishedEvent
LocalParticipant onVideoTrackPublicationFailed LocalVideoTrackPublicationFailedEvent
RemoteDataTrack onStringMessage RemoteDataTrackStringMessageEvent
RemoteDataTrack onBufferMessage RemoteDataTrackBufferMessageEvent
RemoteParticipant onAudioTrackDisabled RemoteAudioTrackEvent
RemoteParticipant onAudioTrackEnabled RemoteAudioTrackEvent
RemoteParticipant onAudioTrackPublished RemoteAudioTrackEvent
RemoteParticipant onAudioTrackSubscribed RemoteAudioTrackSubscriptionEvent
RemoteParticipant onAudioTrackSubscriptionFailed RemoteAudioTrackSubscriptionFailedEvent
RemoteParticipant onAudioTrackUnpublished RemoteAudioTrackEvent
RemoteParticipant onAudioTrackUnsubscribed RemoteAudioTrackSubscriptionEvent
RemoteParticipant onDataTrackPublished RemoteDataTrackEvent
RemoteParticipant onDataTrackSubscribed RemoteDataTrackSubscriptionEvent
RemoteParticipant onDataTrackSubscriptionFailed RemoteDataTrackSubscriptionFailedEvent
RemoteParticipant onDataTrackUnpublished RemoteDataTrackEvent
RemoteParticipant onDataTrackUnsubscribed RemoteDataTrackSubscriptionEvent
RemoteParticipant onVideoTrackDisabled RemoteVideoTrackEvent
RemoteParticipant onVideoTrackEnabled RemoteVideoTrackEvent
RemoteParticipant onVideoTrackPublished RemoteVideoTrackEvent
RemoteParticipant onVideoTrackSubscribed RemoteVideoTrackSubscriptionEvent
RemoteParticipant onVideoTrackSubscriptionFailed RemoteVideoTrackSubscriptionFailedEvent
RemoteParticipant onVideoTrackUnpublished RemoteVideoTrackEvent
RemoteParticipant onVideoTrackUnsubscribed RemoteVideoTrackSubscriptionEvent
Room onConnectFailure RoomConnectFailureEvent
Room onConnected Room
Room onDisconnected RoomDisconnectedEvent
Room onParticipantConnected RoomParticipantConnectedEvent
Room onParticipantDisconnected RoomParticipantDisconnectedEvent
Room onReconnected Room
Room onReconnecting RoomReconnectingEvent
Room onRecordingStarted Room
Room onRecordingStopped Room

开发与贡献

有兴趣贡献?我们欢迎合并请求!请参阅 贡献 指南。

贡献者

HomeX - Home Repairs Made Easy

GitHub

查看 Github