What synchronization protocol is there between the master and slave? PTP?

The hardware timestamps are taken on radio interfaces? Do you know what is the timestamping point for these packets, and if that could explain a fundamental delay asymmetry?

Yes, I am using ptp4l and phc2sys.

The hardware timestamps are taken on Intel I210 NICs. Effectively, the 5G system is completely unaware that it is transmitting PTP packets. The master is a standard Ubuntu PC with an Intel I210 NIC. The slave is the same. The NICs connect to a modem which is the interface to the 5G system, but the timestamping happens at the NIC.

There is fundamental delay asymmetry in the 5G system, I'm just trying to measure it. meanPathDelay from linuxptp is an average of the path delays from master->slave and slave->master. I want to find the individual path delays from master->slave, then from slave->master using isochron. The master->slave latency measurement I'm taking using isochron looks very similar to the meanPathDelay from linuxptp.

Here is linuxptp's meanPathDelay:

and here is isochron master->slave latency:

Does isochron adjust any of its values based on what the offset between master/slave is? I'm guessing even if it did, it can't do that when I use the -o flag. So, is my assumption that I need to adjust isochron measurements by the Time Error correct?

Also, I tried doing this over a simple wireline network (no 5G, just master directly connected to a slave over Ethernet). I'm also seeing negative measurements, but this time, the trend looks reasonable, but obviously latency can't be negative:

Weirdly, I'm also seeing negative meanPathDelay measurements coming from linuxptp:

This might be an issue with the igb driver. Any ideas what might be causing negative/bizarre latencies?

Does isochron adjust any of its values based on what the offset between master/slave is? I'm guessing even if it did, it can't do that when I use the -o flag. So, is my assumption that I need to adjust isochron measurements by the Time Error correct?

No adjustment being done, and no adjustment should be necessary. The RX - TX should be the one-way path delay.

The igb driver however does adjust the reported timestamps.
For TX: https://elixir.bootlin.com/linux/latest/source/drivers/net/ethernet/intel/igb/igb_ptp.c#L959
And for RX: https://elixir.bootlin.com/linux/latest/source/drivers/net/ethernet/intel/igb/igb_ptp.c#L1024

No adjustment being done, and no adjustment should be necessary. The RX - TX should be the one-way path delay.

Isn't that only true when the two devices are synchronized to 0 ns (i.e. the master and slave are on the exact same time base)?
Realistically, downlink rx-tx = offset path_delay, so in order to get the path delay you have to subtract the current offset, especially in a network where ptp4l isn't able to synchronize the devices very well.

Reading the igb_ptp code, it looks like they just adjust it to account for the MAC to PHY delay? (amount of time it takes for the frame to be sent out onto the wire.) I could be wrong. I have seen online that some people have encountered negative path delays while using the igb driver.

Nonetheless, I'm not sure the adjustment made by the igb driver is causing for those drastic changes in latency measurement by isochron, but it's possible.

Not for the drastic fluctuations, no (since the adjustment value is constant, the error comes from somewhere else). Just for the negative part of it.

To see how, let's refer to any diagram of the Pdelay calculation, and apply (asymmetric) timestamp corrections:
https://blog.meinbergglobal.com/wp-content/uploads/2013/09/peer-to-peer-messages1-1.jpg

(t2 - t1) (t4 - t3) / 2 = Pdelay_1
(t2 B_rx - t1 - A_tx t4 A_rx - t3 - B_tx) / 2 = Pdelay_2

Pdelay_2 - Pdelay_1 = (B_rx - A_tx A_rx - B_tx) / 2
Assuming both stations are the same (i210): A_tx = B_tx, and A_rx = B_rx
Pdelay_2 - Pdelay_1 = A_rx - A_tx

In our case, A_rx is IGB_I210_RX_LATENCY_1000 (448) and A_tx is IGB_I210_TX_LATENCY_1000 (178).
So Pdelay_2 - Pdelay_1 is 270 ns.

If the uncorrected Pdelay is smaller than 270 ns, then the corrected one can as well be negative.

Isn't that only true when the two devices are synchronized to 0 ns?

Yeah, I was assuming precise sync.

If the uncorrected Pdelay is smaller than 270 ns, then the corrected one can as well be negative.

Makes sense. So, there's still the drastic fluctuations I'm seeing in the uplink latency. Weirdly enough, this doesn't always happen. Sometimes, I see completely reasonable uplink and downlink latencies, but this is causing me to doubt whether those measurements are legitimate (before and after adjustment by Time Error).

I do think it's worth noting that I am also measuring latency in 3 ways. First, I am measuring pdelay from linuxptp (that's the average of uplink and downlink latencies), uplink latency from isochron, and downlink latency from isochron.

The downlink latency from isochron, pdelay, and PPS are nearly identical to one another in terms of trend. The confounding variable here is the fact that they are all based on the PTP HW Clock (PHC). I would expect that taking an average of the adjusted uplink latency, and adjusted downlink latency would yield the linuxptp pdelay value, or something near it, But because the downlink latency is always drastically different than the uplink latency, the average of the two is nowhere near the linuxptp pdelay. Because of the asymmetry inherent in 5G networks, the uplink and downlink are completely expected to differ, but their average should be the pdelay.

This is probably out of your area of expertise, so my main concern directed toward you is: do you have a theory as to why the uplink latency measured by isochron seems so wildly incorrect (i.e. negative values after adjustment, and drastic fluctuations)?

I mentioned earlier how I can't use VLAN interfaces due to limitations with my network, which prevents me from using the TAPRIO traffic shaper. Would this make my results meaningless, since isochron expects to have a different traffic class than PTP? Since I'm using this over a 5G network, the HW Timestamp latency is really the only thing I care much about. MAC latencies are negligible compared to the latency over the air.

What I'm doing right now is running isochron send and receive on both machines, and calculating the latency between HW Timestamps using R - T on both machines.

I'm thinking the apparent negative latency can be caused by PTP, which expects the path delay to be symmetric.
But do you actually need to run PTP? You have that out-of-band mechanism of measuring the time error between the uncorrected clocks through PPS, so just use isochron with --omit-sync and collect raw timestamps, then correct one of them using the interpolated PPS time error at that point. The remaining component should be the unidirectional path delay.

Correlating the 2 streams of data might be a problem.

Would this make my results meaningless, since isochron expects to have a different traffic class than PTP?

It depends on what you want to measure. The program doesn't necessarily expect to run alone on a traffic class.

do you have a theory as to why the uplink latency measured by isochron seems so wildly incorrect (i.e. negative values after adjustment, and drastic fluctuations)?

I am absolutely confused by the latency figures you've posted being so low (<= 2.5 ns), so there is some physical phenomenon I'm not understanding. But delay asymmetry breaks a lot of the math behind PTP and I simply don't know what to expect.