ffmpeg - 使用 ffmpeg 在节点 js 服务器中处理 WebRTC RTC 流

Question

我正在尝试构建一个视频聊天应用程序，在其中创建 RTCPeerConnection 并创建报价并将 SDP 保存在文件中。我希望我发送到服务器的 SDP 应该发送到像 Nginx RTMP 这样的 RTMP 服务器。这是我得到的 SDP 报价

v=0
o=- 3688975056307578818 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1 2
a=msid-semantic: WMS AqaeHB0L8drTd5r0qQnzCSeYVf4bHFaVqfrq
m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 0 8 106 105 13 110 112 113 126
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:Xtid
a=ice-pwd:jE3iBRpWqFaIN3UJVOAh0G/1
a=ice-options:trickle
a=fingerprint:sha-256 48:E4:36:A6:24:66:F6:40:0F:93:9C:AB:C9:93:DF:C7:0F:D1:21:F5:9E:F7:FA:A8:58:84:1F:68:A1:61:B6:0F
a=setup:actpass
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:5 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:6 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=sendrecv
a=msid:AqaeHB0L8drTd5r0qQnzCSeYVf4bHFaVqfrq 284f316d-6c5d-4283-af4d-86d44803807d
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:103 ISAC/16000
a=rtpmap:104 ISAC/32000
a=rtpmap:9 G722/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:106 CN/32000
a=rtpmap:105 CN/16000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:112 telephone-event/32000
a=rtpmap:113 telephone-event/16000
a=rtpmap:126 telephone-event/8000
a=ssrc:2317617486 cname:fpKjO3/hiHYYBw7w
a=ssrc:2317617486 msid:AqaeHB0L8drTd5r0qQnzCSeYVf4bHFaVqfrq 284f316d-6c5d-4283-af4d-86d44803807d
a=ssrc:2317617486 mslabel:AqaeHB0L8drTd5r0qQnzCSeYVf4bHFaVqfrq
a=ssrc:2317617486 label:284f316d-6c5d-4283-af4d-86d44803807d
m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 102 122 127 121 125 107 108 109 124 120 123 119 114 115 116
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:Xtid
a=ice-pwd:jE3iBRpWqFaIN3UJVOAh0G/1
a=ice-options:trickle
a=fingerprint:sha-256 48:E4:36:A6:24:66:F6:40:0F:93:9C:AB:C9:93:DF:C7:0F:D1:21:F5:9E:F7:FA:A8:58:84:1F:68:A1:61:B6:0F
a=setup:actpass
a=mid:1
a=extmap:14 urn:ietf:params:rtp-hdrext:toffset
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:13 urn:3gpp:video-orientation
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:12 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:11 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:8 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07
a=extmap:9 http://www.webrtc.org/experiments/rtp-hdrext/color-space
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:5 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:6 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=sendrecv
a=msid:AqaeHB0L8drTd5r0qQnzCSeYVf4bHFaVqfrq 7106bf9f-3f79-4f24-959b-b82b603a7acc
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=96
a=rtpmap:98 VP9/90000
a=rtcp-fb:98 goog-remb
a=rtcp-fb:98 transport-cc
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 nack
a=rtcp-fb:98 nack pli
a=fmtp:98 profile-id=0
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=98
a=rtpmap:100 VP9/90000
a=rtcp-fb:100 goog-remb
a=rtcp-fb:100 transport-cc
a=rtcp-fb:100 ccm fir
a=rtcp-fb:100 nack
a=rtcp-fb:100 nack pli
a=fmtp:100 profile-id=2
a=rtpmap:101 rtx/90000
a=fmtp:101 apt=100
a=rtpmap:102 H264/90000
a=rtcp-fb:102 goog-remb
a=rtcp-fb:102 transport-cc
a=rtcp-fb:102 ccm fir
a=rtcp-fb:102 nack
a=rtcp-fb:102 nack pli
a=fmtp:102 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f
a=rtpmap:122 rtx/90000
a=fmtp:122 apt=102
a=rtpmap:127 H264/90000
a=rtcp-fb:127 goog-remb
a=rtcp-fb:127 transport-cc
a=rtcp-fb:127 ccm fir
a=rtcp-fb:127 nack
a=rtcp-fb:127 nack pli
a=fmtp:127 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42001f
a=rtpmap:121 rtx/90000
a=fmtp:121 apt=127
a=rtpmap:125 H264/90000
a=rtcp-fb:125 goog-remb
a=rtcp-fb:125 transport-cc
a=rtcp-fb:125 ccm fir
a=rtcp-fb:125 nack
a=rtcp-fb:125 nack pli
a=fmtp:125 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
a=rtpmap:107 rtx/90000
a=fmtp:107 apt=125
a=rtpmap:108 H264/90000
a=rtcp-fb:108 goog-remb
a=rtcp-fb:108 transport-cc
a=rtcp-fb:108 ccm fir
a=rtcp-fb:108 nack
a=rtcp-fb:108 nack pli
a=fmtp:108 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42e01f
a=rtpmap:109 rtx/90000
a=fmtp:109 apt=108
a=rtpmap:124 H264/90000
a=rtcp-fb:124 goog-remb
a=rtcp-fb:124 transport-cc
a=rtcp-fb:124 ccm fir
a=rtcp-fb:124 nack
a=rtcp-fb:124 nack pli
a=fmtp:124 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=4d0032
a=rtpmap:120 rtx/90000
a=fmtp:120 apt=124
a=rtpmap:123 H264/90000
a=rtcp-fb:123 goog-remb
a=rtcp-fb:123 transport-cc
a=rtcp-fb:123 ccm fir
a=rtcp-fb:123 nack
a=rtcp-fb:123 nack pli
a=fmtp:123 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=640032
a=rtpmap:119 rtx/90000
a=fmtp:119 apt=123
a=rtpmap:114 red/90000
a=rtpmap:115 rtx/90000
a=fmtp:115 apt=114
a=rtpmap:116 ulpfec/90000
a=ssrc-group:FID 967462980 1884395933
a=ssrc:967462980 cname:fpKjO3/hiHYYBw7w
a=ssrc:967462980 msid:AqaeHB0L8drTd5r0qQnzCSeYVf4bHFaVqfrq 7106bf9f-3f79-4f24-959b-b82b603a7acc
a=ssrc:967462980 mslabel:AqaeHB0L8drTd5r0qQnzCSeYVf4bHFaVqfrq
a=ssrc:967462980 label:7106bf9f-3f79-4f24-959b-b82b603a7acc
a=ssrc:1884395933 cname:fpKjO3/hiHYYBw7w
a=ssrc:1884395933 msid:AqaeHB0L8drTd5r0qQnzCSeYVf4bHFaVqfrq 7106bf9f-3f79-4f24-959b-b82b603a7acc
a=ssrc:1884395933 mslabel:AqaeHB0L8drTd5r0qQnzCSeYVf4bHFaVqfrq
a=ssrc:1884395933 label:7106bf9f-3f79-4f24-959b-b82b603a7acc
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=ice-ufrag:Xtid
a=ice-pwd:jE3iBRpWqFaIN3UJVOAh0G/1
a=ice-options:trickle
a=fingerprint:sha-256 48:E4:36:A6:24:66:F6:40:0F:93:9C:AB:C9:93:DF:C7:0F:D1:21:F5:9E:F7:FA:A8:58:84:1F:68:A1:61:B6:0F
a=setup:actpass
a=mid:2
a=sctp-port:5000
a=max-message-size:262144

这是 ffmpeg 命令

ffmpeg -protocol_whitelist rtp,udp,file -loglevel trace -analyzeduration 300M -probesize 300M -i test.sdp -c:v copy -c:a aac -ar 16k -ac 1 -preset ultrafast -tune zerolatency rtmp://127.0.0.1/live/1234

也在尝试这个

ffmpeg -loglevel debug -protocol_whitelist file,crypto,udp,rtp -re -vcodec libvpx -acodec opus -i test.sdp -vcodec libx264 -acodec aac -y output.mp4

FFMpeg 给出这样的错误

ffmpeg version 4.2.2 Copyright (c) 2000-2019 the FFmpeg developers
  built with Apple clang version 11.0.0 (clang-1100.0.33.17)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/4.2.2_2 --enable-shared --enable-pthreads --enable-version3 --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libmp3lame --enable-libopus --enable-librubberband --enable-libsnappy --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-libsoxr --enable-videotoolbox --disable-libjack --disable-indev=jack
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Splitting the commandline.
Reading option '-loglevel' ... matched as option 'loglevel' (set logging level) with argument 'debug'.
Reading option '-protocol_whitelist' ... matched as AVOption 'protocol_whitelist' with argument 'file,crypto,udp,rtp'.
Reading option '-re' ... matched as option 're' (read input at native frame rate) with argument '1'.
Reading option '-vcodec' ... matched as option 'vcodec' (force video codec ('copy' to copy stream)) with argument 'libvpx'.
Reading option '-acodec' ... matched as option 'acodec' (force audio codec ('copy' to copy stream)) with argument 'opus'.
Reading option '-i' ... matched as input url with argument 'test.sdp'.
Reading option '-vcodec' ... matched as option 'vcodec' (force video codec ('copy' to copy stream)) with argument 'libx264'.
Reading option '-acodec' ... matched as option 'acodec' (force audio codec ('copy' to copy stream)) with argument 'aac'.
Reading option '-y' ... matched as option 'y' (overwrite output files) with argument '1'.
Reading option 'output.mp4' ... matched as output url.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option loglevel (set logging level) with argument debug.
Applying option y (overwrite output files) with argument 1.
Successfully parsed a group of options.
Parsing a group of options: input url test.sdp.
Applying option re (read input at native frame rate) with argument 1.
Applying option vcodec (force video codec ('copy' to copy stream)) with argument libvpx.
Applying option acodec (force audio codec ('copy' to copy stream)) with argument opus.
Successfully parsed a group of options.
Opening an input file: test.sdp.
[NULL @ 0x7f949000ac00] Opening 'test.sdp' for reading
[sdp @ 0x7f949000ac00] Format sdp probed with size=2048 and score=50
[sdp @ 0x7f949000ac00] audio codec set to: opus
[sdp @ 0x7f949000ac00] audio samplerate set to: 48000
[sdp @ 0x7f949000ac00] audio channels set to: 2
[sdp @ 0x7f949000ac00] audio codec set to: opus
[sdp @ 0x7f949000ac00] audio samplerate set to: 16000
[sdp @ 0x7f949000ac00] audio channels set to: 1
[sdp @ 0x7f949000ac00] audio codec set to: opus
[sdp @ 0x7f949000ac00] audio samplerate set to: 32000
[sdp @ 0x7f949000ac00] audio channels set to: 1
[sdp @ 0x7f949000ac00] audio codec set to: adpcm_g722
[sdp @ 0x7f949000ac00] audio samplerate set to: 8000
[sdp @ 0x7f949000ac00] audio channels set to: 1
[sdp @ 0x7f949000ac00] audio codec set to: pcm_mulaw
[sdp @ 0x7f949000ac00] audio samplerate set to: 8000
[sdp @ 0x7f949000ac00] audio channels set to: 1
[sdp @ 0x7f949000ac00] audio codec set to: pcm_alaw
[sdp @ 0x7f949000ac00] audio samplerate set to: 8000
[sdp @ 0x7f949000ac00] audio channels set to: 1
[sdp @ 0x7f949000ac00] audio codec set to: pcm_alaw
[sdp @ 0x7f949000ac00] audio samplerate set to: 32000
[sdp @ 0x7f949000ac00] audio channels set to: 1
[sdp @ 0x7f949000ac00] audio codec set to: pcm_alaw
[sdp @ 0x7f949000ac00] audio samplerate set to: 16000
[sdp @ 0x7f949000ac00] audio channels set to: 1
[sdp @ 0x7f949000ac00] audio codec set to: (null)
[sdp @ 0x7f949000ac00] audio samplerate set to: 8000
[sdp @ 0x7f949000ac00] audio channels set to: 1
[sdp @ 0x7f949000ac00] audio codec set to: (null)
[sdp @ 0x7f949000ac00] audio samplerate set to: 48000
[sdp @ 0x7f949000ac00] audio channels set to: 1
[sdp @ 0x7f949000ac00] audio codec set to: (null)
[sdp @ 0x7f949000ac00] audio samplerate set to: 32000
[sdp @ 0x7f949000ac00] audio channels set to: 1
[sdp @ 0x7f949000ac00] audio codec set to: (null)
[sdp @ 0x7f949000ac00] audio samplerate set to: 16000
[sdp @ 0x7f949000ac00] audio channels set to: 1
[sdp @ 0x7f949000ac00] audio codec set to: (null)
[sdp @ 0x7f949000ac00] audio samplerate set to: 8000
[sdp @ 0x7f949000ac00] audio channels set to: 1
[sdp @ 0x7f949000ac00] video codec set to: vp8
    Last message repeated 20 times
[udp @ 0x7f948fc07200] bind failed: Address already in use
[AVIOContext @ 0x7f948fe1c4c0] Statistics: 5985 bytes read, 0 seeks
test.sdp: Invalid data found when processing input

谁能指出我在这里做错了什么？还是我走在正确的道路上？请帮忙！

标签： ffmpegwebrtcsdp

Answer 1

不幸的是（今天）这不起作用，ffmpeg 将来可能会添加 WebRTC 支持！

• WebRTC 需要 Offer/Answer 交换。所以ffmpeg需要生成一个返回SDP（Answer）。
• ffmpeg 没有 ICE 实现。
• ffmpeg 不知道做 DTLS -> SRTP

为此，WebRTC -> RTMP您可以使用它来执行 WebRTC，然后像 [this] （https://github.com/pion/example-webrtc-applications/blob/master/twitch/main.go#L139-L140）一样发送到 ffmpeg。您可以对此进行很多优化。它通过 stdin 执行 mkv 的唯一原因是它可以移植到 Windows。如果你只做 Linux，你可以做多个管道并直接发送 H264/Opus 并为自己节省大量代码。

你也可以使用 GStreamer 来做webrtc -> rtmpsink！