In audio and video streaming, supplemental enhancement information (SEI) is the text data inserted into the audio and video bitstream to convey extra information, which can be received in accurate synchronization with the related audio and video content.
You can use SEI for various purposes. For example:
For more details about SEI, see How to understand and use supplementation enhancement information.
Before you begin, make sure you complete the following:
Create a project in ZEGOCLOUD Admin Console and get the AppID and AppSign of your project.
Refer to the Quick Start doc to complete the SDK integration and basic function implementation.
The following diagram shows the API call sequence of sending and receiving SEI messages.
On the stream publisher side, follow these steps to send SEI messages:
Call the createEngineWithProfile
method to create a ZegoExpressEngine
instance.
Call the loginRoom
method to log in to a room.
Call the startPublishingStream
method to start publishing a stream.
After the stream publishing starts successfully, call the sendSEI
method to send SEI messages when needed.
On the stream subscriber side, follow these steps to receive SEI messages:
createEngineWithProfile
method to create a ZegoExpressEngine
instance.ZegoEventHandler
instance, and override the onPlayerRecvSEI
method to implement your SEI message processing logic.setEventHandler
method with the ZegoEventHandler
instance created in the previous step passed in as the eventHandler
parameter.loginRoom
method to log in to a room.startPlayingStream
method to start playing a stream.onPlayerRecvSEI
callback to process the message.By default, the ZEGO Express SDK uses a ZEGO-defined SEI type (NALU Type = 6, Payload Type = 243) for packaging SEI messages. This SEI type isn't defined in the SEI types of the SDK's video codes and doesn't mix up with the existing SEI data of the source video file.
But if you use a third-party decoder like FFmpeg, the SEI messages with the ZEGO-defined SEI type can't be decoded correctly. To solve this issue, you need to call the setSEIConfig
method to change the SEI type to the UserUnregister type (NALU Type = 6, Payload Type = 5).
As SEI data with Payload Type 5 can also be generated by the video encoder or may exist in the source video file, when you send SEI with Payload Type 5, you need a mechanism to differentiate your SEI data from the SEI data generated by the video encoder itself. For this purpose, you can configure a UUID (16 bytes in length) as an SEI filtering keyword for the ZegoExpressEngine
. On the stream publisher side, pass "UUID + message content" as the buffer to the sendSEI
method when sending SEI messages. On the stream subscriber side, the SDK only throws out those SEI messages that start with the configured UUID. If no UUID is configured, the SDK throws out all received SEI messages.
Function prototype:
/// Sets the supplemental enhancement information (SEI) type
///
/// The SEI type must be set before stream publishing starts.
///
/// @param config: The SEI configuration object. By default, the ZEGO Express SDK uses the ZEGO-defined SEI type to package SEI messages.
-(void)setSEIConfig:(ZegoSEIConfig *)config;
Sample code:
// Configure the SDK to use the UserUnregister SEI type (NALU Type = 6, Payload Type = 5).
ZegoSEIConfig *seiConfig = [[ZegoSEIConfig alloc] init];
seiConfig.type = ZegoSEITypeUserUnregister;
[self.engine setSEIConfig:seiConfig];
// Set up the advancedConfig property to configure a UUID for filtering SEI messages. With this setup, the SDK on the stream subscriber side only throws out those SEI messages of which the first 12 bytes match the configured UUID.
ZegoEngineConfig *engineConfig = [[ZegoEngineConfig alloc] init];
// The first 12 bytes of SEI messages by the stream subscribers through the [onPlayerRecvSEI] callback are always "zegozegozego".
engineConfig.advancedConfig = @{@"unregister_sei_filter": @"zegozegozego"};
[ZegoExpressEngine setEngineConfig:engineConfig];
// Start publishing a stream.
[self.engine startPublishingStream:@"STREAM_ID"];
On the stream publisher side, after the stream publishing starts successfully, you can call the sendSEI
method to send SEI messages when needed.
Funcation prototype:
/// Sends supplemental enhancement information (SEI)
///
/// You can use this method to send extra text information alongside the audio and video stream being published for various purposes.
/// For example, you can use SEI to include video layout information in a video stream or send lyrics text in synchronization with the audio and video content.
/// After the stream publisher sends an SEI message, the stream subscribers can receive it by listening for the [onPlayerRecvSEI] callback.
/// SEI messages are packaged as part of the bitstream of the audio/video frames, which can be lost because of network problems. To solve such potential issues, you can send the same SEI message several times with a limited frequency.
/// Frequency limit: No more than 30 times per second.
/// The maximum length of the SEI data is 4096 bytes.
///
/// @param data: SEI data
- (void)sendSEI:(NSData *)data;
Sample code:
ZegoEngineProfile *profile = [ZegoEngineProfile new];
// The AppID value you get from the ZEGO Admin console.
profile.appID = appID;
// Use the general scenario.
profile.scenario = ZegoScenarioGeneral;
// Create a ZegoExpressEngine instance and set eventHandler to [self]. If eventHandler is set to [nil], no callback will be received. You can set up the event handler later by calling the [-setEventHandler:] method.
self.engine = [ZegoExpressEngine createEngineWithProfile:profile eventHandler:self];
// Log in to a room.
[self.engine loginRoom:roomID user:[ZegoUser userWithUserID:userID userName:userName]];
// Start publishing a stream.
[self.engine startPublishingStream:publishStreamID];
// Implement other business logic.
...;
// Send SEI messages when needed.
char *str = "1234567\0";
[self.manager sendSEI:[NSData dataWithBytes:str length:7 ]];
On the stream subscriber side, after the stream playback starts successfully, the SDK triggers the onPlayerRecvSEI
when an SEI message is received.
Function prototype:
/// The callback triggered when a supplemental enhancement information (SEI) message is received
///
/// After the stream playback starts successfully, the SDK triggers this callback when an SEI message is received.
/// If a subscriber subscribes to the audio stream only, it can't receive any SEI messages.
///
/// @param data: SEI content.
/// @param streamID: Stream ID.
- (void)onPlayerRecvSEI:(NSData *)data streamID:(NSString *)streamID;
Sample code:
@implementation xx
// Listen for the callback that is triggered when SEI messages are received.
- (void)onPlayerRecvSEI:(NSData *)data streamID:(NSString *)streamID {
// Implement your SEI processing logic. For example, present the related UI components.
// ...;
}
// Override other callback methods.
...
@end
ZegoEngineProfile *profile = [ZegoEngineProfile new];
// The AppID value you get from the ZEGO Admin console.
profile.appID = appID;
// Use the general scenario.
profile.scenario = ZegoScenarioGeneral;
// Create a ZegoExpressEngine instance and set eventHandler to [self]. If eventHandler is set to [nil], no callback will be received. You can set up the event handler later by calling the [-setEventHandler:] method.
self.engine = [ZegoExpressEngine createEngineWithProfile:profile eventHandler:self];
// Set up the event handler delegate.
[self.engine setEventHandler:self];
// Log in to a room.
[self.engine loginRoom:roomID user:[ZegoUser userWithUserID:userID userName:userName]];
// Start playing a stream.
[self.engine startPlayingStream:self.firstStreamID canvas:firstPlayCanvas];
// Implement other business logic.
...;