时间戳

首页 > 解决方案 > Gstreamer 从 USB 源播放原始 h.264 流（时间戳问题）

timestamp - Gstreamer 从 USB 源播放原始 h.264 流（时间戳问题）

问题描述

我正在处理通过 USB 输出原始 h.264 视频流的实时视频源（无人机无线视频接收器）。我的目标是将它集成到具有 GStreamer 管道的 Android 中的 QGroundStation 中。

我已将接收到的 USB 数据的一部分转储到一个文件中，使用以下命令可以使用 vlc 完美播放该文件：

vlc -c dump.bin --demux h264

但是，如果我使用这个 GStreamer 管道播放它，播放速度太高（如 x10）

gst-launch-1.0.exe filesrc location="dump.bin" ! h264parse ! avdec_h264 ! autovideosink

我正在使用 appsrc 将 USB 数据推送到 QGroundControl 管道中。视频播放，但是丢了很多帧，gstreamer 抱怨丢包，因为帧太晚了。

[...] 
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.404134207)
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.403025291)
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.401385832)
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.400435290)
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.399607540)
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.398911040)
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.398131998)
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.397308623)
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.396620290)
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.395761040)
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.395125498)
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.394197123)
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.393461831)
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.392803831)
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.391983373)
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.391033998)
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.389664914)
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.388862831)
    W QGroundControl Daily: GStreamerAPILog: <h264parse53> broken/invalid nal Type: 1 Slice, Size: 38639 will be dropped
    W QGroundControl Daily: GStreamerAPILog: <h264parse53> broken/invalid nal Type: 1 Slice, Size: 37460 will be dropped
    W QGroundControl Daily: GStreamerAPILog: <h264parse53> broken/invalid nal Type: 1 Slice, Size: 46566 will be dropped
    W QGroundControl Daily: GStreamerAPILog: <h264parse53> broken/invalid nal Type: 1 Slice, Size: 36055 will be dropped
    W QGroundControl Daily: GStreamerAPILog: <h264parse53> broken/invalid nal Type: 1 Slice, Size: 43397 will be dropped
[...]

在仔细检查我的转储后，我意识到流缺少 pts 和 dts 信息（这在基线 h.264 流中似乎很常见）

ffprobe -show_frames dump.bin

[PACKET]
codec_type=video
stream_index=0
pts=N/A
pts_time=N/A
dts=N/A
dts_time=N/A
duration=48000
duration_time=0.040000
convergence_duration=N/A
convergence_duration_time=N/A
size=1843
pos=0
flags=K_
[/PACKET]
[PACKET]
codec_type=video
stream_index=0
pts=N/A
pts_time=N/A
dts=N/A
dts_time=N/A
duration=48000
duration_time=0.040000
convergence_duration=N/A
convergence_duration_time=N/A
size=16851
pos=1843
flags=K_
[/PACKET]

但显然，持续时间信息在那里。

USB 端点读取 512 字节的块（由于 USB 高速最大。批量端点的有效负载大小），并且一些传输更小（400+ 字节长）。我无法检测 NAL 的开始/结束，因为它是一个不透明的连续字节流。（视频/x-h264，流格式=（字符串）字节流，对齐=无）

因此，我构建了一个 appsrc 来将视频流推送到管道，并尝试像这样盲目地为缓冲区添加时间戳：

void _startFeed(GstElement *source, guint size, gpointer pContext) {
(void)pContext;

GstBuffer *pBuffer;
GstFlowReturn ret;
GstMapInfo map;
guint8 *pRaw;

GstClock *pClk = gst_element_get_clock(gPipeline);
GstClockTime absolute = gst_clock_get_time(pClk);
GstClockTime base = gst_element_get_base_time(gPipeline);
gst_object_unref(pClk);

pBuffer = gst_buffer_new_and_alloc(bufferSize);

// Timestamp the buffers
GST_BUFFER_PTS(pBuffer) = absolute - base;
GST_BUFFER_DTS(pBuffer) = GST_BUFFER_PTS(pBuffer);

gst_buffer_map(pBuffer, &map, GST_MAP_WRITE);

pRaw = (guint8 *)map.data;

fifo_pull(pRaw, size);  // This pulls up to 'size' bytes from the USB FIFO and copies it to pRaw

gst_buffer_unmap(pBuffer, &map);

g_signal_emit_by_name(gAppSrc, "push-buffer", pBuffer, &ret);

gst_buffer_unref(pBuffer);
}

...但仍然没有运气...

通过使用以下管道将 h.264 流编码为 RTP 有效负载，然后使用指定目标帧速率的 caps 过滤器对其进行解码，我取得了有限的成功：

gst-launch-1.0.exe filesrc location="dump.bin" ! video/x-h264, stream-format=(string)byte-stream, alignment=none ! h264parse ! rtph264pay ! rtpjitterbuffer ! rtph264depay ! video/x-h264, stream-format=byte-stream, alignment=nal, framerate=(fraction)30/1 ! h264parse ! avdec_h264 ! autovideosink

我可以将它构建到 C++ 中的 QGroundControl 中，但我认为这不是正确的方法，我不应该对目标帧速率做出任何假设，因为在这种情况下它是 30 fps，但它可能会动态变化。

所以，我的问题是：

更新：：

标签： timestampusbgstreamerh.264raw

解决方案

我使用了一个 h.264 流分析器并意识到这个特定的流在 SPS NAL 中没有 VUI 信息（因此没有解析器可以用来计算 PTS 的 time_tick 或 time_scale 信息）。

对我有用的是在接收器上设置属性 "sync" = false 以在帧到达时立即渲染它们。不理想，但我可以忍受。

推荐阅读