Monitor streaming quality

Introduction

ZEGOCLOUD’s SDKs provide the ability to monitor the streaming quality, user status, video resolution, and status of the streams forwarded to the CDN (Content Delivery Network), set up the callback for the first frame, and receive the SEI (Supplemental Enhancement Information) during stream publishing and stream playing. After stream publishing or stream playing is started, the SDK triggers the corresponding s callback, and client applications can obtain real-time streaming data by listening for these callbacks.

Basic concepts:

• Stream publishing: The process of the client app capturing and transmitting audio and video streams to the ZEGOCLOUD Real-Time Audio and Video Cloud.
• Stream playing: The process of the client app receiving and playing audio and video streams from the ZEGOCLOUD Real-Time Audio and Video Cloud.

Prerequisites

Before you monitor the streaming information, make sure you complete the following:

• A project has been created in ZEGOCLOUD Console and applied for a valid AppID and AppSign. For details, please refer to Console - How to view project information .
• ZEGO Express SDK has been integrated into the project to implement basic real-time audio and video functions. For details, please refer to Integrate the SDK and Implement a basic video call.

Monitor the quality of stream publishing

To monitor the quality of the stream publishing, listen for the onPublisherQualityUpdate callback. After stream publishing is started, this callback will be triggered every 3 seconds to provide the stream publishing quality data.

You can obtain stream publishing quality data through this callback to monitor the health of stream publishing in real time and display the upstreaming network quality on your application's UI.

// You can focus on specific quality attributes and report them to your business server, or focus on a specific field of an object for giving friendly notes for users.
// The parameter 'quality' has many attributes. If you do not want to handle every one of them, you can just focus on the 'level' attribute, a comprehensive quality indicator calculated by the SDK based on other quality attributes.
- (void)onPublisherQualityUpdate:(ZegoPublishStreamQuality *)quality streamID:(NSString *)streamID {
}

The stream publishing quality attributes in details

The stream publishing quality includes the audio and video capture frame rate, bitrate, RTT, packet loss rate, and others of capturing and encoding stages of the stream publishing process.

The ZegoPublishStreamQuality object includes quality attributes of different stages of the stream publishing process. You can find the detailed attribute definitions in this section:

Quality attributes of audio and video capturing

The following attributes reflect the stream publishing quality of audio and video capturing, which is close to the user's subjective perception of the local preview.

• audioCaptureFPS: The audio capture frame rate (fps).
• videoCaptureFPS: The video capture frame rate (fps).

Quality attributes of video encoding

The stream publishing quality attributes of video encoding of the encoding stage is as follow:

• videoEncodeFPS: The target output frame rate (fps) of video encoding.

Quality attributes of stream transmission

The stream publishing quality attributes below reflect the quality of stream transmission (sending), which is related to the encoding bitrate and the current network quality.

• audioSendFPS: The actual audio transmission frame rate (fps).
• audioKBPS: The actual audio transmission bitrate (kbps).
• videoSendFPS: The actual video transmission frame rate (fps).
• videoKBPS: The actual video transmission bitrate (kbps).
• rtt: The rount-trip time (ms) from the client device to the ZEGO server.
• packetLostRate: The client-side upstreaming packet loss rate.

Quality attributes of the total number of bytes

The stream publishing quality attributes below reflect the total number of bytes sent.

• totalSendBytes: The total number of bytes sent, including audio, video, SEI.
• audioSendBytes: The number of audio bytes sent.
• videoSendBytes: The number of video bytes sent.

Quality attributes of encoding information

The stream publishing quality attributes below reflect the encoding information:

• videoCodecID: The video codec.
• isHardwareEncoder: Whether to enable hardware encoding.

Quality attributes of published stream quality level

The parameter quality has many attributes. If you do not want to handle every one of them, you can just focus on the level attribute, a comprehensive upstreaming network qualityindicator calculated by the ZegoExpressEngine based on other quality attributes.

The following table describes the level fields:

Monitor the quality of stream playing

To monitor the quality of the stream playing, listen for the onPlayerQualityUpdate callback. After stream playing is started, this callback will be triggered every 3 seconds to provide the stream playing quality data.

- (void)onPlayerStateUpdate:(ZegoPlayerState)state errorCode:(int)errorCode extendedData:(NSDictionary *)extendedData streamID:(NSString *)streamID {
     // You can focus on specific quality attributes and report them to your business server, or focus on a specific field of an object for giving friendly notes for users.
     // The parameter 'quality' has many attributes. If you do not want to handle every one of them, you can just focus on the 'level' attribute, a comprehensive quality indicator calculated by the SDK based on other quality attributes.
}

The stream playing quality attributes in details

The stream playing quality includes the audio and video frame rate, bitrate, delay, packet loss rate, and others of receiving, decoding, and rendering stages of the stream playing process.

The ZegoPlayStreamQuality object includes quality attributes of different stages of the stream playing process. You can find the detailed attribute definitions in this section:

Quality attributes of stream receiving

The stream playing quality attributes below reflect the quality of stream receiving, which is related to the quality of stream transmission (sending) and the current network quality.

• audioRecvFPS: The actual audio receiving frame rate (fps).
• audioDejitterFPS: The audio dejitter frame rate (f/s).
• audioKBPS: The actual audio receiving bitrate (kbps).
• audioBreakRate: The actual received audio break rate (number of breaks / every 10 seconds).
• videoRecvFPS: The actual video receiving frame rate (fps).
• videoDejitterFPS: The video dejitter frame rate (f/s).
• videoKBPS: The actual video receiving bitrate (kbps).
• videoBreakRate: The actual received video break rate (number of breaks / every 10 seconds).
• packetLostRate: The client-side downstreaming packet loss rate, in percentage, 0.0 - 1.0.
• rtt: The round-trip time (ms) from the client device to the ZEGO server.
• avTimestampDiff: The difference between the video timestamp and the audio timestamp, used to reflect the synchronization of audio and video, in milliseconds. This value is less than 0 means the number of milliseconds that the video leads the audio, greater than 0 means the number of milliseconds that the video lags the audio, and 0 means no difference. When the absolute value is less than 200, it can basically be regarded as synchronized audio and video, when the absolute value is greater than 200 for 10 consecutive seconds, it can be regarded as abnormal.
• peerToPeerDelay: The delay from peer to peer, in milliseconds.
• peerToPeerPacketLostRate: The packet loss rate from peer to peer, in percentage, 0.0 - 1.0.

Quality attributes of stream rendering

The stream playing attributes below reflect the streaming quality at the stream rendering stage, which is close to the viewer's subjective perception. They are affected by the audio/video codec, and the values may be lower than the actual audio/video receiving frame rates.

• audioRenderFPS: The audio rendering frame rate (fps).
• videoRenderFPS: The video rendering frame rate (fps).
• delay: The delay after the data is received by the local end, in milliseconds.

Quality attributes of the total number of bytes

The stream publishing quality attributes below reflect the total number of bytes received.

• totalRecvBytes: The total number of bytes received, including audio, video, SEI.
• audioRecvBytes: The number of audio bytes received.
• videoRecvBytes: The number of video bytes received

Quality attributes of decoding information

The stream playing quality attributes below reflect the decoding information:

• videoCodecID: The video codec.
• isHardwareDecode: Whether to enable hardware decoding.

Monitor the status of stream publishing/playing

Callback for updates on stream publishing status

After stream publishing starts, if the status changes, the SDK sends out the event notification through the onPublisherStateUpdate callback. To monitor the status of stream publishing, listen for this callback.

- (void)onPublisherStateUpdate:(ZegoPublisherState)state errorCode:(int)errorCode extendedData:(NSDictionary *)extendedData streamID:(NSString *)streamID {
        // In this callback, when the [state] is [ZegoPublisherStateNoPublish] and the [errorCode] is not 0, indicates the stream publishing is failed. And the SDK won't retry to publish the stream. At this time, user can set a corresponding UI to indicate the current state.  
        // In this callback, when the [state] is [ZegoPublisherStatePublishRequesting] and the [errorCode] is not 0, indicates the stream publishing retry is ongoing. But if the stream publishing retry still fails after the default reconnection duration, it won't try again and will send out a failure notification.
}

You can determine the user's network status during stream publishing based on the state property.

The following table describes the value of the state property and the corresponding user status:

The parameter extendedData provides extended information for status updates. If you are using the ZEGO CDN, the key of this parameter is flv_url_list, rtmp_url_list, and hls_url_list, which correspond to the stream playing URL of FLV, RTMP, and HLS protocols respectively.

Callback for updates on stream playing status

After stream playing starts, if the status changes, the SDK sends out the event notification through the onPlayerStateUpdate callback. To monitor the status of stream playing, listen for this callback.

- (void)onPlayerStateUpdate:(ZegoPlayerState)state errorCode:(int)errorCode extendedData:(NSDictionary *)extendedData streamID:(NSString *)streamID {
        // In this callback, when the [state] is [ZegoPlayerStateNoPlay] and the [errorCode] is not 0, indicates the stream playing is failed. And the SDK won't retry to play the stream. At this time, user can set a corresponding UI to indicate the current state. 
        // In this callback, when the [state] is [ZegoPlayerStatePlayRequesting] and the [errorCode] is not 0, indicates the stream playing retry is ongoing. But if the stream playing retry still fails after the default reconnection duration, it won't try again and will send out a failure notification.
}

You can determine the user's network status during stream playing based on the state property.

The following table describes the value of the state property and the corresponding user status:

Monitor the first frame of audio and video

Callback on capturing the first frame of audio

To receive the event notification when the first frame of audio is captured, listen for the onPublisherCapturedAudioFirstFrame callback.

After stream publishing starts, the SDK sends out the event notification through this callback when the first frame of audio is captured.

- (void)onPublisherCapturedAudioFirstFrame {

}

Callback on capturing the first frame of video

To receive the event notification when the first frame of video is captured, listen for the onPublisherCapturedVideoFirstFrame callback.

After stream publishing starts, the SDK sends out the event notification through this callback when the first frame of video is captured.

- (void)onPublisherCapturedVideoFirstFrame:(ZegoPublishChannel)channel {

}

Callback on receiving the first frame of audio

To receive the event notification when the first frame of audio is received, listen for the onPlayerRecvAudioFirstFrame callback.

After stream playing starts, the SDK sends out the event notification through this callback when the first frame of audio is received.

- (void)onPlayerRecvAudioFirstFrame:(NSString *)streamID {
}

Callback on receiving the first frame of video

To receive the event notification when the first frame of video is received, listen for the onPlayerRecvVideoFirstFrame callback.

After stream playing starts, the SDK sends out the event notification through this callback when the first frame of video is received.

- (void)onPlayerRecvVideoFirstFrame:(NSString *)streamID {
}

Callback on rendering the first frame of video

To receive the event notification when the first frame of video is rendered, listen for the onPlayerRenderVideoFirstFrame callback.

After stream playing starts, the SDK sends out the event notification through this callback when the first frame of video is received and has been rendered.

- (void)onPlayerRenderVideoFirstFrame:(NSString *)streamID
}

Monitor the changes on video resolution

Callback for the changes on the captured video resolution

To receive the event notification when the captured video size changes, listen for the onPublisherVideoSizeChanged callback.

After stream publishing starts, the SDK sends out the event notification through this callback when the resolution of the captured video changes.

- (void)onPublisherCapturedVideoFirstFrame:(ZegoPublishChannel)channel {
}

Callback for the changes on the played video resolution

To receive the event notification when the played video resolution changes, listen for the onPlayerVideoSizeChanged callback.

After stream playing starts, the SDK sends out the event notification through this callback when the resolution of the played video changes. You can adjust the video display based on the final resolution of the video streams.

- (void)onPublisherVideoSizeChanged:(CGSize)size channel:(ZegoPublishChannel)channel {
}

Monitor the status of the stream forwarding via CDN

Callback for updates on the status of CDN URL

To receive event notification when the specified CDN URL changes (adds new URL or removes existing URL), listen for the onPublisherRelayCDNStateUpdate callback.

After the ZEGO RTC Server forwards the stream to CDN, the SDK sends out event notification when the status of streams forwarded to CDN changes, for example, the stream forwarding stops or the forwarding operation retries.

- (void)onPublisherRelayCDNStateUpdate:(NSArray<ZegoStreamRelayCDNInfo *> *)infoList streamID:(NSString *)streamID {
}

The stream forwarding attributes in details

The stream forwarding information includes the CDN URL, forwarding status, the cause of stream forwarding status, and when the status changes.

The ZegoStreamRelayCDNInfo object includes stream forwarding attributes. You can find the detailed attribute definitions in this section:

The following table describes the value of the state property:

The following table describes the value of the updateReason property: