twilio_programmable_video

用于 Twilio Programmable Video 的 Flutter 插件,它使您能够构建实时视频通话应用程序(WebRTC)。此 Flutter 插件是一个社区维护的项目,用于 Twilio Programmable Video,并非由 Twilio 维护。如果您有任何问题,请提交一个 issue,而不是联系支持。

此软件包目前正在开发中,不应用于生产应用程序。在达到 v1.0.0 之前,我们无法保证当前 API 实现会在版本之间保持不变。

示例

请查看我们提供的、包含在此插件中的全面的 示例

Twilio Programmable Video Example

加入社区

如果您有任何疑问或问题,请加入我们的 Discord

常见问题

在创建新的 issue 之前,请先阅读 常见问题解答

支持的平台

  • Android
  • iOS
  • Web (预发布)

入门

先决条件

在使用插件之前,您需要确保为您的项目准备好所有设置。

Android

为了使此插件在 Android 上正常工作,您需要修改一些文件。

权限

打开您 android/app/src/main 目录下的 AndroidManifest.xml 文件,并添加以下设备权限

...
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.CAMERA"/>
<uses-permission android:name="android.permission.BLUETOOTH"/>
<uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS"/>
    ...
Proguard

将以下行添加到您的 android/app/proguard-rules.pro 文件中。

-keep class tvi.webrtc.** { *; }
-keep class com.twilio.video.** { *; }
-keep class com.twilio.common.** { *; }
-keepattributes InnerClasses

另外,请不要忘记在您的 android/app/build.gradle 文件中引用此 proguard-rules.pro

android {

    ...

    buildTypes {

        release {

            ...

            minifyEnabled true
            useProguard true

            proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'

        }
    }
}

iOS

为了使此插件在 iOS 上正常工作,您需要修改一些文件。

权限

打开您 ios/Runner 目录下的 Info.plist 文件,并添加以下权限

...
<key>NSCameraUsageDescription</key>
<string>Your message to user when the camera is accessed for the first time</string>
<key>NSMicrophoneUsageDescription</key>
<string>Your message to user when the microphone is accessed for the first time</string>
<key>io.flutter.embedded_views_preview</key>
<true/>
...

打开您 ios 目录下的 Podfile 文件,并添加以下权限

...
post_install do |installer|
  installer.pods_project.targets.each do |target|
    flutter_additional_ios_build_settings(target)
    target.build_configurations.each do |config|
      config.build_settings['GCC_PREPROCESSOR_DEFINITIONS'] ||= [
        '$(inherited)',
        'PERMISSION_CAMERA=1',
        'PERMISSION_MICROPHONE=1',
        # Add other permissions required by your app
      ]

    end
  end
end
...

有关示例,请查看示例应用程序的 Podfile

设置最低 iOS 版本为 11
  1. 在 Xcode 中,打开您应用程序 ios 文件夹下的 Runner.xcworkspace
  2. 要查看您应用的设置,请在 Xcode 项目导航器中选择 **Runner** 项目。然后,在主视图侧边栏中,选择 **Runner** 目标。
  3. 选择 **General** 选项卡。
  4. 在 **Deployment Info** 部分,将 Target 设置为 iOS 11。
后台模式

为了允许在应用程序运行时在后台保持与房间的连接,您必须在 Capabilities 项目设置页面选择 Audio、AirPlay 和 Picture in Picture 背景模式。有关更多信息,请参阅 Twilio 文档

Web

为了使此插件在 Web 上工作,您需要在 web 目录下的 index.html 中添加一个包含 twilio-video JavaScript 软件包的脚本标签。

一种简单的方法是在 index.html<head> 中添加以下行

<!DOCTYPE html>
<html>
  <head>
    ...

    <script src="//media.twiliocdn.com/sdk/js/video/releases/2.14.0/twilio-video.min.js"></script>
  </head>
  ....
</html>

版本(在本例中为 2.14.0)将在连接时进行检查。然后,插件将检查

  • 主版本号是否与插件中定义的相同。
  • 次版本号是否小于或等于插件中定义的。

如果未能通过任何这些检查,它将在运行时引发 UnsupportedError 错误。

连接到房间

在您的 Flutter 应用程序中,调用 TwilioProgrammableVideo.connect() 连接到房间。连接后,您可以与其他连接到房间的参与者(Participants)发送和接收音频和视频流。

Room _room;
final Completer<Room> _completer = Completer<Room>();

void _onConnected(Room room) {
  print('Connected to ${room.name}');
  _completer.complete(_room);
}

void _onConnectFailure(RoomConnectFailureEvent event) {
  print('Failed to connect to room ${event.room.name} with exception: ${event.exception}');
  _completer.completeError(event.exception);
}

Future<Room> connectToRoom() async {
  // Retrieve the camera source of your choosing
  var cameraSources = await CameraSource.getSources();
  var cameraCapturer = CameraCapturer(
    cameraSources.firstWhere((source) => source.isFrontFacing),
  );

  var connectOptions = ConnectOptions(
    accessToken,
    roomName: roomName,                   // Optional name for the room
    region: region,                       // Optional region.
    preferredAudioCodecs: [OpusCodec()],  // Optional list of preferred AudioCodecs
    preferredVideoCodecs: [H264Codec()],  // Optional list of preferred VideoCodecs.
    audioTracks: [LocalAudioTrack(true)], // Optional list of audio tracks.
    dataTracks: [
      LocalDataTrack(
        DataTrackOptions(
          ordered: ordered,                      // Optional, Ordered transmission of messages. Default is `true`.
          maxPacketLifeTime: maxPacketLifeTime,  // Optional, Maximum retransmit time in milliseconds. Default is [DataTrackOptions.defaultMaxPacketLifeTime]
          maxRetransmits: maxRetransmits,        // Optional, Maximum number of retransmitted messages. Default is [DataTrackOptions.defaultMaxRetransmits]
          name: name                             // Optional
        ),                                // Optional
      ),
    ],                                    // Optional list of data tracks
    videoTracks: [LocalVideoTrack(true, cameraCapturer)], // Optional list of video tracks.
  );
  _room = await TwilioProgrammableVideo.connect(connectOptions);
  _room.onConnected.listen(_onConnected);
  _room.onConnectFailure.listen(_onConnectFailure);
  return _completer.future;
}

连接到房间时,您必须提供 访问令牌

加入房间

如果您想加入一个已知已存在的房间,处理方式与创建房间完全相同:只需将房间名称传递给 connect 方法。进入房间后,您将收到每个成功加入的参与者的 RoomParticipantConnectedEvent。查询 room.remoteParticipants getter 将返回已加入房间的任何现有参与者。

Room _room;
final Completer<Room> _completer = Completer<Room>();

void _onConnected(Room room) {
  print('Connected to ${room.name}');
  _completer.complete(_room);
}

void _onConnectFailure(RoomConnectFailureEvent event) {
  print('Failed to connect to room ${event.room.name} with exception: ${event.exception}');
  _completer.completeError(event.exception);
}

Future<Room> connectToRoom() async {
  // Retrieve the camera source of your choosing
  var cameraSources = await CameraSource.getSources();
  var cameraCapturer = CameraCapturer(
    cameraSources.firstWhere((source) => source.isFrontFacing),
  );

  var connectOptions = ConnectOptions(
    accessToken,
    roomName: roomName,
    region: region,                       // Optional region.
    preferAudioCodecs: [OpusCodec()],     // Optional list of preferred AudioCodecs
    preferVideoCodecs: [H264Codec()],     // Optional list of preferred VideoCodecs.
    audioTracks: [LocalAudioTrack(true)], // Optional list of audio tracks.
    dataTracks: [
      LocalDataTrack(
        DataTrackOptions(
          ordered: ordered,                      // Optional, Ordered transmission of messages. Default is `true`.
          maxPacketLifeTime: maxPacketLifeTime,  // Optional, Maximum retransmit time in milliseconds. Default is [DataTrackOptions.defaultMaxPacketLifeTime]
          maxRetransmits: maxRetransmits,        // Optional, Maximum number of retransmitted messages. Default is [DataTrackOptions.defaultMaxRetransmits]
          name: name                             // Optional
        ),                                // Optional
      ),
    ],                                    // Optional list of data tracks
    videoTracks([LocalVideoTrack(true, cameraCapturer)]), // Optional list of video tracks.
  );
  _room = await TwilioProgrammableVideo.connect(connectOptions);
  _room.onConnected.listen(_onConnected);
  _room.onConnectFailure.listen(_onConnectFailure);
  return _completer.future;
}

设置本地媒体

您可以通过以下方式从设备的麦克风或摄像头捕获本地媒体

// Create an audio track.
var localAudioTrack = LocalAudioTrack(true);

// Retrieve the camera source of your choosing
var cameraSources = await CameraSource.getSources();
var cameraCapturer = CameraCapturer(
  cameraSources.firstWhere((source) => source.isFrontFacing),
);

// Create a video track.
var localVideoTrack = LocalVideoTrack(true, cameraCapturer);

// Getting the local video track widget.
// This can only be called after the TwilioProgrammableVideo.connect() is called.
var widget = localVideoTrack.widget();

作为仅发布参与者连接

目前无法作为仅发布参与者连接。

处理远程参与者

处理已连接的参与者

当您加入一个房间时,可能已经存在其他参与者。您可以在 Room.onConnected 监听器被调用时,使用 room.remoteParticipants getter 来检查已存在的参与者。

// Connect to a room.
var room = await TwilioProgrammableVideo.connect(connectOptions);

room.onConnected((Room room) {
  print('Connected to ${room.name}');
});

room.onConnectFailure((RoomConnectFailureEvent event) {
    print('Failed connecting, exception: ${event.exception.message}');
});

room.onDisconnected((RoomDisconnectEvent event) {
  print('Disconnected from ${event.room.name}');
});

room.onRecordingStarted((Room room) {
  print('Recording started in ${room.name}');
});

room.onRecordingStopped((Room room) {
  print('Recording stopped in ${room.name}');
});

// ... Assume we have received the connected callback.

// After receiving the connected callback the LocalParticipant becomes available.
var localParticipant = room.localParticipant;
print('LocalParticipant ${room.localParticipant.identity}');

// Get the first participant from the room.
var remoteParticipant = room.remoteParticipants[0];
print('RemoteParticipant ${remoteParticipant.identity} is in the room');

处理参与者连接事件

当参与者连接或断开与您连接的房间的连接时,您将通过事件监听器收到通知。这些事件有助于您的应用程序跟踪加入或离开房间的参与者。

// Connect to a room.
var room = await TwilioProgrammableVideo.connect(connectOptions);

room.onParticipantConnected((RoomParticipantConnectedEvent event) {
  print('Participant ${event.remoteParticipant.identity} has joined the room');
});

room.onParticipantDisconnected((RoomParticipantDisconnectedEvent event) {
  print('Participant ${event.remoteParticipant.identity} has left the room');
});

显示远程参与者的 Widget

要查看远程参与者发送的视频轨道,我们需要将它们的 Widget 添加到树中。

room.onParticipantConnected((RoomParticipantConnectedEvent roomEvent) {
  // We can respond when the Participant adds a VideoTrack by adding the widget to the tree.
  roomEvent.remoteParticipant.onVideoTrackSubscribed((RemoteVideoTrackSubscriptionEvent event) {
    var mirror = false;
    _widgets.add(event.remoteParticipant.widget(mirror));
  });
});

使用 DataTrack API

DataTrack API 允许您创建一个 DataTrack 通道,该通道可用于向订阅了数据的零个或多个接收者发送低延迟消息。

目前,开始使用 DataTrack 的唯一方法是在 连接到房间 时在 ConnectOptions 中指定它。

连接到房间后,您必须等到收到 LocalDataTrackPublishedEvent 才能开始向该轨道发送数据。您可以在连接到房间后使用 Room.onConnected 监听器开始监听此事件。

// Connect to a room.
var room = await TwilioProgrammableVideo.connect(connectOptions);

room.onConnected((Room room) {
  // Once connected to the room start listening for the moment the LocalDataTrack gets published to the room.
  room.localParticipant.onDataTrackPublished.listen(_onLocalDataTrackPublished);
});

  // Once connected to the room start listening for the moment the LocalDataTrack gets published to the room.
  event.room.localParticipant.onDataTrackPublished.listen(_onLocalDataTrackPublished);
});

void _onLocalDataTrackPublished(LocalDataTrackPublishedEvent event) {
  // This event contains a localDataTrack you can use to send data.
  event.localDataTrackPublication.localDataTrack.send('Hello world');
}

如果您想接收来自 RemoteDataTrack 的数据,您必须在 RemoteParticipant 开始发布该轨道并且您已订阅它后,开始监听该轨道。

// Connect to a room.
var room = await TwilioProgrammableVideo.connect(connectOptions);

room.onParticipantConnected((RoomParticipantConnectedEvent event) {
  // A participant connected, now you can start listening to RemoteParticipant events
  event.remoteParticipant.onDataTrackSubscribed.listen(_onDataTrackSubscribed)
});

void _onDataTrackSubscribed(RemoteDataTrackSubscriptionEvent event) {
  final dataTrack = event.remoteDataTrackPublication.remoteDataTrack;
  dataTrack.onMessage.listen(_onMessage);
}

void _onMessage(RemoteDataTrackStringMessageEvent event) {
  print('onMessage => ${event.remoteDataTrack.sid}, ${event.message}');
}

请记住,您将不会收到在您开始监听之前发送的消息。

参与房间

显示摄像头预览

就像 Twilio 一样,我们完全理解您想在进入房间前看起来很棒。

// Provide a `create` function to the `LocalVideoTrack` class that will trigger initialization at the native layer.
await localVideoTrack?.create(); 

//Add a publishTrack method to LocalParticipants to allow for publishing LocalVideoTracks as needed.
await localVideoTrack?.publish();

断开与房间的连接

您可以断开与您当前参与的房间的连接。其他参与者将收到 RoomParticipantDisconnectedEvent

// To disconnect from a Room, we call:
await room.disconnect();

// This results in a call to Room#onDisconnected
room.onDisconnected((RoomDisconnectEvent event) {
  print('Disconnected from ${event.room.name}');
});

房间重新连接

房间重新连接是由信令或媒体重新连接事件触发的。

/// Exception will be either TwilioException.signalingConnectionDisconnectedException or TwilioException.mediaConnectionErrorException
room.onReconnecting((RoomReconnectingEvent event) {
  print('Reconnecting to room ${event.room.name}, exception = ${event.exception.message}');
});

room.onReconnected((Room room) {
  print('Reconnected to room ${room.name}');
});

配置音频、视频输入和输出设备

利用控制输入和输出设备的能力,可以为最终用户构建更好的体验。

选择特定的视频输入

CameraCapturer 类用于从给定的 CameraSourceLocalVideoTrack 提供视频帧。

// Share your camera.
var cameraSources = await CameraSource.getSources();
var cameraCapturer = CameraCapturer(
  cameraSources.firstWhere((source) => source.isFrontFacing),
);
var localVideoTrack = LocalVideoTrack(true, cameraCapturer);

// Render camera to a widget (only after connect event).
var mirror = true;
var widget = localVideoTrack.widget(mirror);
_widgets.add(widget);

// Switch the camera source.
var cameraSources = await CameraSource.getSources();
var cameraSource = cameraSources.firstWhere((source) => source.isBackFacing);
await cameraCapturer.switchCamera(cameraSource);
await primaryVideoView.setMirror(cameraSource.isBackFacing);

选择特定的音频输出

使用 TwilioProgrammableVideo 类,您可以指定音频是通过耳机、扬声器还是可用的蓝牙音频设备路由。

笔记

如果 speakerphoneEnabledbluetoothPreferred 都为 true,则如果可用,将使用蓝牙音频设备,否则音频将通过扬声器路由。

// Route audio through speaker
await TwilioProgrammableVideo.setAudioSettings(speakerphoneEnabled: true, bluetoothPreferred: false);

// Route audio through headset
await TwilioProgrammableVideo.setAudioSettings(speakerphoneEnabled: false, bluetoothPreferred: false);

// Use Bluetooth if available, otherwise use the headset.
await TwilioProgrammableVideo.setAudioSettings(speakerphoneEnabled: false, bluetoothPreferred: true);

// Use Bluetooth if available, otherwise use the speaker.
await TwilioProgrammableVideo.setAudioSettings(speakerphoneEnabled: true, bluetoothPreferred: true);

笔记

调用 setAudioSettings 后,Android 和 iOS 实现将监听路由更改,并努力确保继续使用应用的音频设置。虽然如此,您可以通过 TwilioProgrammableVideo 类监听此类更改。

TwilioProgrammableVideo.onAudioNotification.listen((event) {
// do things.
});

要禁用音频设置管理和路由更改观察,请使用 TwilioProgrammableVideo 类调用 disableAudioSettings

await TwilioProgrammableVideo.disableAudioSettings();

播放音频文件以提供丰富的用户体验

iOS

为了在您使用此插件时播放音频文件,我们推荐使用 ocarina 插件(v0.1.2 及更高版本)。

此推荐是在调查了 Flutter 生态系统中可用于此功能的插件,以及能够与本插件良好配合的插件之后提出的。

观察到的主要问题是,在 iOS 上,大多数插件会修改 AVAudioSession 模式,将其设置为仅播放模式,从而阻止视频通话录制音频。

iOS 上音频文件播放的第二个问题是,操作系统优先使用 VoiceProcessingIO Audio Unit,导致在使用此 AudioUnit 时,其他音频源的播放音量大大降低。为了解决此问题,我们提供了自定义的 AVAudioEngineDevice,插件用户可以使用如下示例进行启用。AVAudioEngineDevice 是为 ocarina 设计的,提供了一个接口,用于将音频文件播放和管理委托给该插件的 AVAudioEngineDevice。它是从 Twilio 的示例改编的。

要启用 AVAudioEngineDevice 的使用,并将 ocarina 的音频文件播放管理委托给它,请按以下方式更新您的 AppDelegate.swiftdidFinishLaunch 方法

  override func application(
    _ application: UIApplication,
    didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?
  ) -> Bool {
    GeneratedPluginRegistrant.register(with: self)

    let audioDevice = AVAudioEngineDevice.getInstance()
    SwiftTwilioProgrammableVideoPlugin.setCustomAudioDevice(
        audioDevice,
        onConnected: audioDevice.onConnected,
        onDisconnected: audioDevice.onDisconnected)
    SwiftOcarinaPlugin.useDelegate(
        load: audioDevice.addMusicNode,
        dispose: audioDevice.disposeMusicNode,
        play: audioDevice.playMusic,
        pause: audioDevice.pauseMusic,
        resume: audioDevice.resumeMusic,
        stop: audioDevice.stopMusic,
        volume: audioDevice.setMusicVolume,
        seek: audioDevice.seekPosition,
        position: audioDevice.getPosition
    )

    return super.application(application, didFinishLaunchingWithOptions: launchOptions)
  }

完成此操作后,您应该可以像往常一样继续使用此插件和 ocarina

Android

从版本 0.11.0 开始,我们也为 Android 提供了与 ocarina 的集成。

此集成的目的是允许基于播放状态的智能管理音频设置和音频焦点。

要获得此集成的优势,请将以下内容添加到您的 MainActivity.kt

    private lateinit var PACKAGE_ID: String

    @RequiresApi(Build.VERSION_CODES.O)
    override fun configureFlutterEngine(@NonNull flutterEngine: FlutterEngine) {
        super.configureFlutterEngine(flutterEngine)
        PACKAGE_ID = applicationContext.packageName
        OcarinaPlugin.addListener(PACKAGE_ID, TwilioProgrammableVideoPlugin.getAudioPlayerEventListener());
    }

    override fun cleanUpFlutterEngine(@NonNull flutterEngine: FlutterEngine) {
        super.cleanUpFlutterEngine(flutterEngine)
        OcarinaPlugin.removeListener(PACKAGE_ID)
    }

启用调试日志

使用 TwilioProgrammableVideo 类,您可以启用插件的原生和 Dart 日志记录。

var nativeEnabled = true;
var dartEnabled = true;
TwilioProgrammableVideo.debug(native: nativeEnabled, dart: dartEnabled);

访问令牌

请记住,您不能使用 TestCredentials 为 programmable-video 生成访问令牌,请使用 LIVE 凭据。

您可以使用 Twilio 控制台中的 测试工具轻松生成访问令牌以开始测试您的代码。但我们建议您设置一个后端来为您生成这些令牌并保护您的 Twilio 凭据。就像我们在 示例应用程序中所做的那样。

事件表

插件当前支持的所有事件的参考表

类型 事件流 事件数据 已实现
LocalParticipant onAudioTrackPublished LocalAudioTrackPublishedEvent
LocalParticipant onAudioTrackPublicationFailed LocalAudioTrackPublicationFailedEvent
LocalParticipant onDataTrackPublished LocalDataTrackPublishedEvent
LocalParticipant onDataTrackPublicationFailed LocalDataTrackPublicationFailedEvent
LocalParticipant onVideoTrackPublished LocalVideoTrackPublishedEvent
LocalParticipant onVideoTrackPublicationFailed LocalVideoTrackPublicationFailedEvent
RemoteDataTrack onStringMessage RemoteDataTrackStringMessageEvent
RemoteDataTrack onBufferMessage RemoteDataTrackBufferMessageEvent
RemoteParticipant onAudioTrackDisabled RemoteAudioTrackEvent
RemoteParticipant onAudioTrackEnabled RemoteAudioTrackEvent
RemoteParticipant onAudioTrackPublished RemoteAudioTrackEvent
RemoteParticipant onAudioTrackSubscribed RemoteAudioTrackSubscriptionEvent
RemoteParticipant onAudioTrackSubscriptionFailed RemoteAudioTrackSubscriptionFailedEvent
RemoteParticipant onAudioTrackUnpublished RemoteAudioTrackEvent
RemoteParticipant onAudioTrackUnsubscribed RemoteAudioTrackSubscriptionEvent
RemoteParticipant onDataTrackPublished RemoteDataTrackEvent
RemoteParticipant onDataTrackSubscribed RemoteDataTrackSubscriptionEvent
RemoteParticipant onDataTrackSubscriptionFailed RemoteDataTrackSubscriptionFailedEvent
RemoteParticipant onDataTrackUnpublished RemoteDataTrackEvent
RemoteParticipant onDataTrackUnsubscribed RemoteDataTrackSubscriptionEvent
RemoteParticipant onVideoTrackDisabled RemoteVideoTrackEvent
RemoteParticipant onVideoTrackEnabled RemoteVideoTrackEvent
RemoteParticipant onVideoTrackPublished RemoteVideoTrackEvent
RemoteParticipant onVideoTrackSubscribed RemoteVideoTrackSubscriptionEvent
RemoteParticipant onVideoTrackSubscriptionFailed RemoteVideoTrackSubscriptionFailedEvent
RemoteParticipant onVideoTrackUnpublished RemoteVideoTrackEvent
RemoteParticipant onVideoTrackUnsubscribed RemoteVideoTrackSubscriptionEvent
Room onConnectFailure RoomConnectFailureEvent
Room onConnected Room
Room onDisconnected RoomDisconnectedEvent
Room onParticipantConnected RoomParticipantConnectedEvent
Room onParticipantDisconnected RoomParticipantDisconnectedEvent
Room onReconnected Room
Room onReconnecting RoomReconnectingEvent
Room onRecordingStarted Room
Room onRecordingStopped Room

开发与贡献

有兴趣贡献吗?我们非常欢迎合并请求!请参阅 贡献 指南。

贡献者

HomeX - Home Repairs Made Easy

GitHub

查看 Github