twilio_programmable_video
用于 Twilio Programmable Video 的 Flutter 插件,它使您能够构建实时视频通话应用程序(WebRTC)。此 Flutter 插件是社区维护的项目,由 Twilio 维护。如果您有任何问题,请提交问题,而不是联系支持。
此包目前正在开发中,不应用于生产应用程序。在我们达到 v1.0.0 之前,我们不能保证当前 API 实现会在版本之间保持不变。
示例
请查看我们在此插件中提供的详尽的 示例。
加入社区
如果您有任何问题或困难,请加入我们的 Discord。
常见问题
在创建新问题之前,请先阅读 常见问题解答。
支持的平台
- Android
- iOS
- Web(查看预发布版本)
入门
先决条件
在开始使用该插件之前,您需要确保您的项目已设置好所有内容。
Android
为了让此插件在 Android 上正常运行,您需要调整一些文件。
权限
打开您 android/app/src/main 目录中的 AndroidManifest.xml 文件,并添加以下设备权限
...
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.CAMERA"/>
<uses-permission android:name="android.permission.BLUETOOTH"/>
<uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS"/>
...
Proguard
将以下行添加到您的 android/app/proguard-rules.pro 文件中。
-keep class tvi.webrtc.** { *; }
-keep class com.twilio.video.** { *; }
-keep class com.twilio.common.** { *; }
-keepattributes InnerClasses
别忘了在您的 android/app/build.gradle 文件中引用此 proguard-rules.pro。
android {
...
buildTypes {
release {
...
minifyEnabled true
useProguard true
proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
}
}
}
iOS
为了让此插件在 iOS 上正常运行,您需要调整一些文件。
权限
打开您 ios/Runner 目录中的 Info.plist 文件,并添加以下权限
...
<key>NSCameraUsageDescription</key>
<string>Your message to user when the camera is accessed for the first time</string>
<key>NSMicrophoneUsageDescription</key>
<string>Your message to user when the microphone is accessed for the first time</string>
<key>io.flutter.embedded_views_preview</key>
<true/>
...
打开您 ios 目录中的 Podfile 文件,并添加以下权限
...
post_install do |installer|
installer.pods_project.targets.each do |target|
flutter_additional_ios_build_settings(target)
target.build_configurations.each do |config|
config.build_settings['GCC_PREPROCESSOR_DEFINITIONS'] ||= [
'$(inherited)',
'PERMISSION_CAMERA=1',
'PERMISSION_MICROPHONE=1',
# Add other permissions required by your app
]
end
end
end
...
有关示例,请查看示例应用程序的 Podfile。
将最低 iOS 版本设置为 11
- 在 Xcode 中,打开您应用程序
ios文件夹中的Runner.xcworkspace。 - 要查看您应用程序的设置,请在 Xcode 项目导航器中选择 Runner 项目。然后,在主视图侧边栏中,选择 Runner 目标。
- 选择 General 选项卡。
- 在 Deployment Info 部分,将 Target 设置为 iOS 11。
后台模式
要允许在应用程序后台运行时保持与房间的连接,您必须从 Capabilities 项目设置页面中选择 Audio、AirPlay 和 Picture in Picture 后台模式。有关更多信息,请参阅 Twilio 文档。
连接到房间
在您的 Flutter 应用程序中,调用 TwilioProgrammableVideo.connect() 来连接到房间。连接后,您可以与其他连接到房间的参与者进行音视频流的发送和接收。
Room _room;
final Completer<Room> _completer = Completer<Room>();
void _onConnected(Room room) {
print('Connected to ${room.name}');
_completer.complete(_room);
}
void _onConnectFailure(RoomConnectFailureEvent event) {
print('Failed to connect to room ${event.room.name} with exception: ${event.exception}');
_completer.completeError(event.exception);
}
Future<Room> connectToRoom() async {
// Retrieve the camera source of your choosing
var cameraSources = await CameraSource.getSources();
var cameraCapturer = CameraCapturer(
cameraSources.firstWhere((source) => source.isFrontFacing),
);
var connectOptions = ConnectOptions(
accessToken,
roomName: roomName, // Optional name for the room
region: region, // Optional region.
preferredAudioCodecs: [OpusCodec()], // Optional list of preferred AudioCodecs
preferredVideoCodecs: [H264Codec()], // Optional list of preferred VideoCodecs.
audioTracks: [LocalAudioTrack(true)], // Optional list of audio tracks.
dataTracks: [
LocalDataTrack(
DataTrackOptions(
ordered: ordered, // Optional, Ordered transmission of messages. Default is `true`.
maxPacketLifeTime: maxPacketLifeTime, // Optional, Maximum retransmit time in milliseconds. Default is [DataTrackOptions.defaultMaxPacketLifeTime]
maxRetransmits: maxRetransmits, // Optional, Maximum number of retransmitted messages. Default is [DataTrackOptions.defaultMaxRetransmits]
name: name // Optional
), // Optional
),
], // Optional list of data tracks
videoTracks: [LocalVideoTrack(true, cameraCapturer)], // Optional list of video tracks.
);
_room = await TwilioProgrammableVideo.connect(connectOptions);
_room.onConnected.listen(_onConnected);
_room.onConnectFailure.listen(_onConnectFailure);
return _completer.future;
}
连接到房间时,您必须提供 访问令牌。
加入房间
如果您想加入一个已知存在的房间,您可以以与创建房间完全相同的方式处理:只需将房间名称传递给 connect 方法即可。进入房间后,您将为每个成功加入的参与者收到一个 RoomParticipantConnectedEvent。查询 room.remoteParticipants getter 将返回已加入房间的任何现有参与者。
Room _room;
final Completer<Room> _completer = Completer<Room>();
void _onConnected(Room room) {
print('Connected to ${room.name}');
_completer.complete(_room);
}
void _onConnectFailure(RoomConnectFailureEvent event) {
print('Failed to connect to room ${event.room.name} with exception: ${event.exception}');
_completer.completeError(event.exception);
}
Future<Room> connectToRoom() async {
// Retrieve the camera source of your choosing
var cameraSources = await CameraSource.getSources();
var cameraCapturer = CameraCapturer(
cameraSources.firstWhere((source) => source.isFrontFacing),
);
var connectOptions = ConnectOptions(
accessToken,
roomName: roomName,
region: region, // Optional region.
preferAudioCodecs: [OpusCodec()], // Optional list of preferred AudioCodecs
preferVideoCodecs: [H264Codec()], // Optional list of preferred VideoCodecs.
audioTracks: [LocalAudioTrack(true)], // Optional list of audio tracks.
dataTracks: [
LocalDataTrack(
DataTrackOptions(
ordered: ordered, // Optional, Ordered transmission of messages. Default is `true`.
maxPacketLifeTime: maxPacketLifeTime, // Optional, Maximum retransmit time in milliseconds. Default is [DataTrackOptions.defaultMaxPacketLifeTime]
maxRetransmits: maxRetransmits, // Optional, Maximum number of retransmitted messages. Default is [DataTrackOptions.defaultMaxRetransmits]
name: name // Optional
), // Optional
),
], // Optional list of data tracks
videoTracks([LocalVideoTrack(true, cameraCapturer)]), // Optional list of video tracks.
);
_room = await TwilioProgrammableVideo.connect(connectOptions);
_room.onConnected.listen(_onConnected);
_room.onConnectFailure.listen(_onConnectFailure);
return _completer.future;
}
设置本地媒体
您可以通过以下方式从设备的麦克风或摄像头捕获本地媒体:
// Create an audio track.
var localAudioTrack = LocalAudioTrack(true);
// Retrieve the camera source of your choosing
var cameraSources = await CameraSource.getSources();
var cameraCapturer = CameraCapturer(
cameraSources.firstWhere((source) => source.isFrontFacing),
);
// Create a video track.
var localVideoTrack = LocalVideoTrack(true, cameraCapturer);
// Getting the local video track widget.
// This can only be called after the TwilioProgrammableVideo.connect() is called.
var widget = localVideoTrack.widget();
连接为仅发布参与者
目前无法连接为仅发布参与者。
处理远程参与者
处理已连接的参与者
当您加入房间时,可能会有参与者已经存在。您可以在调用 Room.onConnected 侦听器时,通过使用 room.remoteParticipants getter 来检查现有参与者。
// Connect to a room.
var room = await TwilioProgrammableVideo.connect(connectOptions);
room.onConnected((Room room) {
print('Connected to ${room.name}');
});
room.onConnectFailure((RoomConnectFailureEvent event) {
print('Failed connecting, exception: ${event.exception.message}');
});
room.onDisconnected((RoomDisconnectEvent event) {
print('Disconnected from ${event.room.name}');
});
room.onRecordingStarted((Room room) {
print('Recording started in ${room.name}');
});
room.onRecordingStopped((Room room) {
print('Recording stopped in ${room.name}');
});
// ... Assume we have received the connected callback.
// After receiving the connected callback the LocalParticipant becomes available.
var localParticipant = room.localParticipant;
print('LocalParticipant ${room.localParticipant.identity}');
// Get the first participant from the room.
var remoteParticipant = room.remoteParticipants[0];
print('RemoteParticipant ${remoteParticipant.identity} is in the room');
处理参与者连接事件
当参与者连接或断开您正在连接的房间时,您将通过事件侦听器收到通知。这些事件可帮助您的应用程序跟踪加入或离开房间的参与者。
// Connect to a room.
var room = await TwilioProgrammableVideo.connect(connectOptions);
room.onParticipantConnected((RoomParticipantConnectedEvent event) {
print('Participant ${event.remoteParticipant.identity} has joined the room');
});
room.onParticipantDisconnected((RoomParticipantDisconnectedEvent event) {
print('Participant ${event.remoteParticipant.identity} has left the room');
});
显示远程参与者的 Widget
要查看远程参与者发送的视频轨道,我们需要将他们的 Widget 添加到树中。
room.onParticipantConnected((RoomParticipantConnectedEvent roomEvent) {
// We can respond when the Participant adds a VideoTrack by adding the widget to the tree.
roomEvent.remoteParticipant.onVideoTrackSubscribed((RemoteVideoTrackSubscriptionEvent event) {
var mirror = false;
_widgets.add(event.remoteParticipant.widget(mirror));
});
});
使用 DataTrack API
DataTrack API 允许您创建一个 DataTrack 通道,该通道可用于向订阅了数据的零个或多个接收者发送低延迟消息。
目前,开始使用 DataTrack 的唯一方法是在 连接到房间 时在 ConnectOptions 中指定它。
连接到房间后,您必须等待收到 LocalDataTrackPublishedEvent,然后才能开始向该轨道发送数据。您可以使用 Room.onConnected 侦听器在连接到房间后开始收听此事件。
// Connect to a room.
var room = await TwilioProgrammableVideo.connect(connectOptions);
room.onConnected((Room room) {
// Once connected to the room start listening for the moment the LocalDataTrack gets published to the room.
room.localParticipant.onDataTrackPublished.listen(_onLocalDataTrackPublished);
});
// Once connected to the room start listening for the moment the LocalDataTrack gets published to the room.
event.room.localParticipant.onDataTrackPublished.listen(_onLocalDataTrackPublished);
});
void _onLocalDataTrackPublished(LocalDataTrackPublishedEvent event) {
// This event contains a localDataTrack you can use to send data.
event.localDataTrackPublication.localDataTrack.send('Hello world');
}
如果您想接收来自 RemoteDataTrack 的数据,您必须在 RemoteParticipant 开始发布该轨道并且您已订阅它之后开始收听该轨道。
// Connect to a room.
var room = await TwilioProgrammableVideo.connect(connectOptions);
room.onParticipantConnected((RoomParticipantConnectedEvent event) {
// A participant connected, now you can start listening to RemoteParticipant events
event.remoteParticipant.onDataTrackSubscribed.listen(_onDataTrackSubscribed)
});
void _onDataTrackSubscribed(RemoteDataTrackSubscriptionEvent event) {
final dataTrack = event.remoteDataTrackPublication.remoteDataTrack;
dataTrack.onMessage.listen(_onMessage);
}
void _onMessage(RemoteDataTrackStringMessageEvent event) {
print('onMessage => ${event.remoteDataTrack.sid}, ${event.message}');
}
请记住,您将不会收到在您开始收听之前发送的消息。
参与房间
显示摄像头预览
就像 Twilio 一样,我们完全理解您希望在进入房间前看起来很棒。
// Provide a `create` function to the `LocalVideoTrack` class that will trigger initialization at the native layer.
await localVideoTrack?.create();
//Add a publishTrack method to LocalParticipants to allow for publishing LocalVideoTracks as needed.
await localVideoTrack?.publish();
断开与房间的连接
您可以断开与您当前参与的房间的连接。其他参与者将收到 RoomParticipantDisconnectedEvent。
// To disconnect from a Room, we call:
await room.disconnect();
// This results in a call to Room#onDisconnected
room.onDisconnected((RoomDisconnectEvent event) {
print('Disconnected from ${event.room.name}');
});
房间重新连接
由于信令或媒体重新连接事件,会触发房间重新连接。
/// Exception will be either TwilioException.signalingConnectionDisconnectedException or TwilioException.mediaConnectionErrorException
room.onReconnecting((RoomReconnectingEvent event) {
print('Reconnecting to room ${event.room.name}, exception = ${event.exception.message}');
});
room.onReconnected((Room room) {
print('Reconnected to room ${room.name}');
});
配置音频、视频输入和输出设备
利用控制输入和输出设备的能力,您可以构建更好的最终用户体验。
选择特定的视频输入
CameraCapturer 类用于从给定的 CameraSource 提供 LocalVideoTrack 的视频帧。
// Share your camera.
var cameraSources = await CameraSource.getSources();
var cameraCapturer = CameraCapturer(
cameraSources.firstWhere((source) => source.isFrontFacing),
);
var localVideoTrack = LocalVideoTrack(true, cameraCapturer);
// Render camera to a widget (only after connect event).
var mirror = true;
var widget = localVideoTrack.widget(mirror);
_widgets.add(widget);
// Switch the camera source.
var cameraSources = await CameraSource.getSources();
var cameraSource = cameraSources.firstWhere((source) => source.isBackFacing);
await cameraCapturer.switchCamera(cameraSource);
await primaryVideoView.setMirror(cameraSource.isBackFacing);
选择特定的音频输出
使用 TwilioProgrammableVideo 类,您可以指定音频是通过耳机、扬声器还是可用的蓝牙音频设备路由。
笔记
如果
speakerphoneEnabled和bluetoothPreferred都为 true,则将在可用时使用蓝牙音频设备,否则音频将通过扬声器路由。
// Route audio through speaker
await TwilioProgrammableVideo.setAudioSettings(speakerphoneEnabled: true, bluetoothPreferred: false);
// Route audio through headset
await TwilioProgrammableVideo.setAudioSettings(speakerphoneEnabled: false, bluetoothPreferred: false);
// Use Bluetooth if available, otherwise use the headset.
await TwilioProgrammableVideo.setAudioSettings(speakerphoneEnabled: false, bluetoothPreferred: true);
// Use Bluetooth if available, otherwise use the speaker.
await TwilioProgrammableVideo.setAudioSettings(speakerphoneEnabled: true, bluetoothPreferred: true);
笔记
调用
setAudioSettings后,Android 和 iOS 实现将监听路由更改,并努力确保继续使用应用的音频设置。虽然是这样,您可以使用TwilioProgrammableVideo类监听此类更改。
TwilioProgrammableVideo.onAudioNotification.listen((event) {
// do things.
});
要禁用音频设置管理和路由更改观察,请使用
TwilioProgrammableVideo类调用disableAudioSettings。
await TwilioProgrammableVideo.disableAudioSettings();
播放音频文件以提供丰富的用户体验
iOS
为了在该插件中使用时播放音频文件,我们推荐使用 ocarina 插件(v0.1.2 及更高版本)。
此建议是在调查了 Flutter 生态系统中提供此功能的可用插件之后得出的,以找出与此插件配合良好的插件。
观察到的其他提供此功能的插件的主要问题是,在 iOS 上,大多数插件会修改 AVAudioSession 模式,将其置于仅播放模式,因此阻止视频通话录制音频。
iOS 上音频文件播放的第二个问题是,操作系统会优先处理 VoiceProcessingIO Audio Unit,导致在使用此 AudioUnit 时,其他音频源的播放音量大大降低。为了解决这个问题,我们提供了自定义的 AVAudioEngineDevice,插件用户可以启用它,并提供如下示例。AVAudioEngineDevice 是从 Twilio 的示例改编而来的,旨在为从该插件到 AVAudioEngineDevice 的音频文件播放和管理提供一个接口。
要启用 AVAudioEngineDevice 的使用,并将音频文件播放管理从 ocarina 委托给它,请按如下方式更新您的 AppDelegate.swift 的 didFinishLaunch 方法。
override func application(
_ application: UIApplication,
didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?
) -> Bool {
GeneratedPluginRegistrant.register(with: self)
let audioDevice = AVAudioEngineDevice.getInstance()
SwiftTwilioProgrammableVideoPlugin.setCustomAudioDevice(
audioDevice,
onConnected: audioDevice.onConnected,
onDisconnected: audioDevice.onDisconnected)
SwiftOcarinaPlugin.useDelegate(
load: audioDevice.addMusicNode,
dispose: audioDevice.disposeMusicNode,
play: audioDevice.playMusic,
pause: audioDevice.pauseMusic,
resume: audioDevice.resumeMusic,
stop: audioDevice.stopMusic,
volume: audioDevice.setMusicVolume,
seek: audioDevice.seekPosition,
position: audioDevice.getPosition
)
return super.application(application, didFinishLaunchingWithOptions: launchOptions)
}
完成此操作后,您应该能够像往常一样继续使用此插件和 ocarina。
Android
从版本 0.11.0 开始,我们还在 Android 上提供了与 ocarina 的集成。
此次集成的目的是允许根据播放状态智能管理音频设置和音频焦点。
要获得此集成的优势,请将以下内容添加到您的 MainActivity.kt。
private lateinit var PACKAGE_ID: String
@RequiresApi(Build.VERSION_CODES.O)
override fun configureFlutterEngine(@NonNull flutterEngine: FlutterEngine) {
super.configureFlutterEngine(flutterEngine)
PACKAGE_ID = applicationContext.packageName
OcarinaPlugin.addListener(PACKAGE_ID, TwilioProgrammableVideoPlugin.getAudioPlayerEventListener());
}
override fun cleanUpFlutterEngine(@NonNull flutterEngine: FlutterEngine) {
super.cleanUpFlutterEngine(flutterEngine)
OcarinaPlugin.removeListener(PACKAGE_ID)
}
启用调试日志记录
使用 TwilioProgrammableVideo 类,您可以启用插件的原生和 Dart 日志记录。
var nativeEnabled = true;
var dartEnabled = true;
TwilioProgrammableVideo.debug(native: nativeEnabled, dart: dartEnabled);
访问令牌
请记住,您无法使用 TestCredentials 为 programmable-video 生成访问令牌,请使用 LIVE 凭据。
您可以在 Twilio 控制台的 测试工具 中轻松生成访问令牌来开始测试您的代码。但我们建议您设置后端来为您生成这些令牌并保护您的 Twilio 凭据。就像我们在 示例应用程序 中所做的那样。
事件表
该插件当前支持的所有事件的参考表
| 类型 | 事件流 | 事件数据 | 已实现 |
|---|---|---|---|
| LocalParticipant | onAudioTrackPublished | LocalAudioTrackPublishedEvent | 是 |
| LocalParticipant | onAudioTrackPublicationFailed | LocalAudioTrackPublicationFailedEvent | 是 |
| LocalParticipant | onDataTrackPublished | LocalDataTrackPublishedEvent | 是 |
| LocalParticipant | onDataTrackPublicationFailed | LocalDataTrackPublicationFailedEvent | 是 |
| LocalParticipant | onVideoTrackPublished | LocalVideoTrackPublishedEvent | 是 |
| LocalParticipant | onVideoTrackPublicationFailed | LocalVideoTrackPublicationFailedEvent | 是 |
| RemoteDataTrack | onStringMessage | RemoteDataTrackStringMessageEvent | 是 |
| RemoteDataTrack | onBufferMessage | RemoteDataTrackBufferMessageEvent | 是 |
| RemoteParticipant | onAudioTrackDisabled | RemoteAudioTrackEvent | 是 |
| RemoteParticipant | onAudioTrackEnabled | RemoteAudioTrackEvent | 是 |
| RemoteParticipant | onAudioTrackPublished | RemoteAudioTrackEvent | 是 |
| RemoteParticipant | onAudioTrackSubscribed | RemoteAudioTrackSubscriptionEvent | 是 |
| RemoteParticipant | onAudioTrackSubscriptionFailed | RemoteAudioTrackSubscriptionFailedEvent | 是 |
| RemoteParticipant | onAudioTrackUnpublished | RemoteAudioTrackEvent | 是 |
| RemoteParticipant | onAudioTrackUnsubscribed | RemoteAudioTrackSubscriptionEvent | 是 |
| RemoteParticipant | onDataTrackPublished | RemoteDataTrackEvent | 是 |
| RemoteParticipant | onDataTrackSubscribed | RemoteDataTrackSubscriptionEvent | 是 |
| RemoteParticipant | onDataTrackSubscriptionFailed | RemoteDataTrackSubscriptionFailedEvent | 是 |
| RemoteParticipant | onDataTrackUnpublished | RemoteDataTrackEvent | 是 |
| RemoteParticipant | onDataTrackUnsubscribed | RemoteDataTrackSubscriptionEvent | 是 |
| RemoteParticipant | onVideoTrackDisabled | RemoteVideoTrackEvent | 是 |
| RemoteParticipant | onVideoTrackEnabled | RemoteVideoTrackEvent | 是 |
| RemoteParticipant | onVideoTrackPublished | RemoteVideoTrackEvent | 是 |
| RemoteParticipant | onVideoTrackSubscribed | RemoteVideoTrackSubscriptionEvent | 是 |
| RemoteParticipant | onVideoTrackSubscriptionFailed | RemoteVideoTrackSubscriptionFailedEvent | 是 |
| RemoteParticipant | onVideoTrackUnpublished | RemoteVideoTrackEvent | 是 |
| RemoteParticipant | onVideoTrackUnsubscribed | RemoteVideoTrackSubscriptionEvent | 是 |
| Room | onConnectFailure | RoomConnectFailureEvent | 是 |
| Room | onConnected | Room | 是 |
| Room | onDisconnected | RoomDisconnectedEvent | 是 |
| Room | onParticipantConnected | RoomParticipantConnectedEvent | 是 |
| Room | onParticipantDisconnected | RoomParticipantDisconnectedEvent | 是 |
| Room | onReconnected | Room | 是 |
| Room | onReconnecting | RoomReconnectingEvent | 是 |
| Room | onRecordingStarted | Room | 是 |
| Room | onRecordingStopped | Room | 是 |
开发与贡献
有兴趣贡献?我们欢迎合并请求!请参阅 贡献 指南。
