WebRTC integration #60
Comments
Welcome back!
Best guess:
Conceptually we have RTP packets coming in (from the RTSP server) and want RTP packets going out (to the WebRTC client). I think there are a few valid approaches:
You're effectively doing 1 now. As long as that's working, great. If you run into problems, we can move on to the other approaches. I'd like to eventually offer 3 as part of retina.
Yeah, I think it'd be great to have examples of how to link Retina with other code: ffmpeg, gstreamer, webrtc, etc. I think ideally we'd skip the retransmit/receive over UDP stage and glue Retina and webrtc together in the same process. |
|
Hi
You are right, the Bunch of --- ---CTRL-C signal---
I also think the thrid one is the best/valid solution to transmit the data, i will try to get some information about how to do that one, sounds like a good functionality to develop to get a more solid rust and video stream knowledge.
I was thinking on something like this, but in rust (with the retina getting the data), what you think about it?. |
Interesting. I was expecting a
Yeah, that'd be very nice. If it's simple, it could be just be a retina example; if it's a full-featured version, it should probably be a separate crate, whether in this repo or otherwise. |
|
Hi.
When i use UDP transport can't get a single
I already wrote to dave @rtspstream, i hope he can help us.
I don't know how to do this, but i will find the way and posting back.
Nice, i will confirm you. As a note, when i use |
|
If you don't mind dropping the audio; I found it was easier to use retina to pop off the raw h264 frames: let frame = match demuxed.next().await {
None => break;
Some(Ok(CodecItem::VideoFrame(frame))) => {
frame
}
Some(Ok(_)) => continue,
Some(Err(err)) => {
break;
}
};This will give you frames in AVC format, but to use them in webrtc you will want to convert them to Annex B. While this involves a memory copy, you will want to inspect the NALs anyways to throwaway any NALUs that come in that can effect streaming. In my case, an amcrest camera I was using was sending SEI NALUs which caused Safari's WebRTC player to throwup. This is the same approach that RTSPtoWeb uses. Using webrtc-rs and h264-reader let mut frame = frame.data.clone();
let mut sps_nal = Bytes::new();
let mut pps_nal = Bytes::new();
while frame.len() > 0 {
use bytes::Buf;
use h264_reader::nal::UnitType;
let sz = frame.get_u32() as usize;
let nal = frame.split_to(sz);
let nal_type = match nal.get(0).and_then(|n| UnitType::for_id(n & 0x1F).ok()) {
Some(n) => n,
None => continue,
};
let data = match nal_type {
UnitType::SliceLayerWithoutPartitioningIdr => {
if !idr {
idr = true;
}
let mut v = Vec::with_capacity(
4 sps_nal.len() 4 pps_nal.len() 4 nal.len(),
);
v.extend_from_slice(&[0, 0, 0, 1]);
v.extend_from_slice(sps_nal.as_ref());
v.extend_from_slice(&[0, 0, 0, 1]);
v.extend_from_slice(pps_nal.as_ref());
v.extend_from_slice(&[0, 0, 0, 1]);
v.extend_from_slice(nal.as_ref());
Bytes::from(v)
}
UnitType::SliceLayerWithoutPartitioningNonIdr => {
if !idr {
continue;
}
let mut v = Vec::with_capacity(4 nal.len());
v.extend_from_slice(&[0, 0, 0, 1]);
v.extend_from_slice(nal.as_ref());
Bytes::from(v)
}
UnitType::SeqParameterSet => {
sps_nal = nal.clone();
continue;
}
UnitType::PicParameterSet => {
pps_nal = nal.clone();
continue;
}
_ => continue,
};
let data_sz = data.len();
let err = video_track
.write_sample(&Sample {
data,
duration: frame_duration,
timestamp: frame_timestamp,
packet_timestamp: packet_timestamp,
..Default::default()
})
.await;
} |
|
@sergiomeneses: Sorry, I initially missed your comment with the Wireshark output, and the full dump links no longer work: "File has been removed."
Retina should be following my habit of either returning errors or logging them, not both at once. If it returns, you can log if yourself with whatever extra context you have higher up the stack. Basically, I prefer fewer, richer log messages over more frequent ones that have to be pieced together to get the complete picture. Anyway, I don't know why the server is returning status 400. If you can post those full captures again, I'd be interested in comparing Retina's failing request with the successful one from VLC to know what went wrong. @nemosupremo: thanks for your working example! Interesting point about Safari not liking the SEI NALs. I imagine that might have been a pain to debug. fwiw, I'm open to adding to Retina a way to request data in say Annex B format with non-VCL NALs stripped out, so you don't have to do that conversion yourself. This is the kind of thing I was imagining using A couple interesting cases you might encounter:
|
You can probably imagine; as the debugging tools in Safari are delightfully unhelpful. The "fix" was just noticing what RTSPToWeb was doing.
Ah, I thought
I haven't seen it in the wild either; and most other code I see online don't handle them. I am currently working on replacing some parts of gstreamer with retina in our application and the cameras we have tested so far stream fine. I'll have to double check. |
Yeah, I went through the same thing a couple times when Safari didn't like Moonfire's
Currently it means it has at least one IDR slice: Line 466 in 33bd057 I think a frame ("access unit" in H.264 terms) is supposed to have either all IDR or all non-IDR NALs (but I could be wrong). There's something called periodic infra refresh, but I think it uses non-IDR NALs with slice type P/B and some type-I macroblocks, then adds in "SEI recovery point" NALs periodically. Not quite sure how that should be handled either. Maybe the SPS/PPS should be (re)sent on the SEI recovery point NAL? Except if Safari needs SEI stripped out, it probably doesn't handle periodic infra refresh well regardless of what the sender does... |
This is a very basic version, as a starting point.
|
I just pushed a commit with a very basic WebRTC proxy example. Improvements welcome! |
|
hey @scottlamb i was a little busy.
I think this is fixed on the
i will do it. Don't know if we can close this as fixed. |
For what it's worth (which might not be much), another example of this would be my rtsp-to-webrtc. It's intended to be part of a larger system I'm creating, but also to be used stand-alone. It can handle multiple streams, and uses a variation on the WISH protocol for signalling between the server and the browser. There's a sample HTML page which can be opened in a browser, too. |
Hey @scottlamb , im here again after some time, do you remember me? (from discussions) ;).
I have some time now to start with my PoC (e.g: rtp to webrtc)
Im using retina to get the rtsp stream (without demuxed) and pasing it through a UDP socket to my webrtc server (example), this partially works (once start the stream suddenly stops without a clear reason, apparently retina stops sending packets).
I am using as a RTSP source stream the rtsp.stream service this.
the retina code is something like this:
I want to get some guideness about if is this the correct path to go, i was reading #58 and he wants to pass the frames using
gstreamerrtph264depay, you answers him about using thedemuxed()of retina to do that.In my case using
demuxed()onsessiondoesn't works, i keep improving this PoC to maybe one day add it to the examples if you think is valuable.Thanks for you work on this awesome crate.
The text was updated successfully, but these errors were encountered: