Convey extra information using SEI

Introduction

In audio and video streaming, supplemental enhancement information (SEI) is the text data inserted into the audio and video bitstream to convey extra information, which can be received in accurate synchronization with the related audio and video content.

You can use SEI for various purposes. For example:

• To include the video layout information in the stream.
• To send lyrics text in synchronization with the audio and video content.
• To send quiz information in live quizzing use cases.

Prerequisites

Before you begin, make sure you complete the following:

• Create a project in ZEGOCLOUD Admin Console and get the AppID and AppSign of your project.

• Refer to the Quick Start doc to complete the SDK integration and basic function implementation.

Implementation process

The following diagram shows the API call sequence of sending and receiving SEI messages.

On the stream publisher side, follow these steps to send SEI messages:

1. Call the createEngine method to create a ZegoExpressEngine instance.

2. Call the loginRoom method to log in to a room.

3. Call the startPublishingStream method to start publishing a stream.

4. After the stream publishing starts successfully, call the sendSEI method to send SEI messages when needed.

On the stream subscriber side, follow these steps to receive SEI messages:

1. Call the createEngine method to create a ZegoExpressEngine instance.
2. Create an IZegoEventHandler instance, and override the onPlayerRecvSEI method to implement your SEI message processing logic.
3. Call the setEventHandler method with the IZegoEventHandler instance created in the previous step passed in as the eventHandler parameter.
4. Call the loginRoom method to log in to a room.
5. Call the startPlayingStream method to start playing a stream.
6. After the stream playback starts successfully, when an SEI message is received, the SDK triggers the onPlayerRecvSEI callback to process the message.

Optional: Set the SEI type

Send SEI messages

On the stream publisher side, after the stream publishing starts successfully, you can call the sendSEI method to send SEI messages when needed.

• Funcation prototype:

  /**
   * Sends supplemental enhancement information (SEI)
   *
   * You can use this method to send extra text information alongside the audio and video stream being published for various purposes. 
   * For example, you can use SEI to include video layout information in a video stream or send lyrics text in synchronization with the audio and video content.
   * After the stream publisher sends an SEI message, the stream subscribers can receive it by listening for the [onPlayerRecvSEI] callback.
   * SEI messages are packaged as part of the bitstream of the audio/video frames, which can be lost because of network problems. To solve such potential issues, you can send the same SEI message several times with a limited frequency. 
   * Frequency limit: No more than 30 times per second.
   * The maximum length of the SEI data is 4096 bytes.
   * @param data: SEI data
   */
  public void sendSEI(byte[] data)

• Sample code:

  /** Define a ZegoExpressEngine object */
  ZegoExpressEngine engine;
  
  ZegoEngineProfile profile = new ZegoEngineProfile();
  /** AppID format: 123456789L */
  profile.appID = appID;
  /** General scenario */
  profile.scenario = ZegoScenario.GENERAL;
  /** Set application object of App */
  profile.application = getApplication();
  /** Create a ZegoExpressEngine instance */
  engine = ZegoExpressEngine.createEngine(profile, null);
  // Log in to a room.
  engine.loginRoom("roomid", new ZegoUser("userid_1"));
  // Start publishing a stream.
  engine.startPublishingStream("streamid");
  // Implement other business logic.
  ...;
  // Send SEI messages when needed.
  engine.sendSEI("12345".getBytes());

Receive SEI messages

On the stream subscriber side, after the stream playback starts successfully, the SDK triggers the onPlayerRecvSEI callback when an SEI message is received.

• Function prototype:

  /**
   * The callback triggered when a supplemental enhancement information (SEI) message is received
   *
   * After the stream playback starts successfully, the SDK triggers this callback when an SEI message is received.
   * If a subscriber subscribes to the audio stream only, it can't receive any SEI messages.
   * @param data: SEI content
   * @param streamID: Stream ID
   */
  public void onPlayerRecvSEI(String streamID, byte[] data){
  
  }

• Sample code:

  /** Define a ZegoExpressEngine object */
  ZegoExpressEngine engine;
  
  ZegoEngineProfile profile = new ZegoEngineProfile();
  /** AppID format: 123456789L */
  profile.appID = appID;
  /** General scenario */
  profile.scenario = ZegoScenario.GENERAL;
  /** Set application object of App */
  profile.application = getApplication();
  /** Create a ZegoExpressEngine instance */
  engine = ZegoExpressEngine.createEngine(profile, null);
  
  // Create an IZegoEventHandler object, and override the onPlayerRecvSEI method to implement your SEI message processing logic.
  IZegoEventHandler handler = new IZegoEventHandler(){
      // Listen for other callbacks.
      ...;
  
      // Listen for the callback that is triggered when an SEI message is received.
      public void onPlayerRecvSEI(String streamID, byte[] data) {
          // Implement your SEI processing logic. For example, present the related UI components.
          ...;
      }
  
  }
  // Set up the event handler.
  engine.setEventHandler(handler);
  // Log in to a room.
  engine.loginRoom("roomid", new ZegoUser("userid_2"));
  // Start playing a stream.
  engine.startPlayingStream("streamid", canvas);
  // Implement other business logic.
  ...;