-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
LTX fd polling is slow in Qemu #3
Comments
I did more debugging and it looks like while the code in kirk is absurdly suboptimal but it may actually work. The polling task calls epoll to check for data availability and read which is dumb but may actually work (which is equivalent to calling read in a loop on O_NONBLOCK socket). It looks like the problem here is that virtio serial connected to fifo in qemu drops data. Normally when someone tries to write into a fifo that has no reader the call blocks, but it looks that this is not true for virtio serial connected to a fifo, where the data are silently dropped unless someone is reading at the other end. I suppose that the easiest solution to fix this is to add support to unix sockets and tcp to kirk, so that we can connect the virtio serial to unix socket that should be able to buffer and retain data. |
And in the end I've managed to actually make things work. And the problem was qemu configuration, if you create a single fifo for a virtio-serial both ends of the fifo are connected to the serial and bytes arrive randomly either to anything that reads the fifo on host or back to the qemu virtio serial. It seems to be nondeterministic and some bytes are simply lost 🤦. To start qemu with virtio device you need to do
To start ltx on the SUT you need
Now to talk to the ltx with kirk on host:
This is enough to run simple commands, however things are awfully slow for some reason and attempts to run a tesuite hangs or end up with errors, so there are some bottlenecks/bugs in the communication. Things seem to work, albeit slow if I add debugging messages to ltx, which hints at problems in kirk deserializer. |
The fact LTX is slow with debug messages is quite normal. Are you sure this is a bug? |
@acerv Try yourself, it's slow and it hangs, either at start or shortly after. I suppose that something is broken in the part that deserializes the messages. |
@metan-ucw what's your qemu version?
|
@acerv Ah, sorry the commandline is incomplete, you have to add |
Sorry but I can't reproduce this problem:
Am I missing something? |
Ah, another missing part in the qemu parameters, you have to pass |
The epoll usage has been removed with #11 |
It seems the real bottle neck is related with data transfer, because testing suites execution is really fast (at the end it's only a commands list run). It could be even a LTX bug, but I need to go deeper into it. If I increase the I also noticed that if LTX is compiled without debug info, the data transfer just doesn't work and kirk stuck waiting for data. This is not something related with kirk, but rather LTX -> Qemu serial port communication. |
It looks like the file descriptor is not registered correctly when ltx is selected in kirk. What it does is to register new epoll instance and a asyncio task that polls that instance. The problem is that there is a second instance of epoll in the asyncio main loop that is used to wait for new events. I suppose that we would have to use low level asyncio add_reader() function to watch the file descriptor however that API allows us to register a reader function that would have to somehow feed data to the corutine parser and I have no idea how to do that.
The text was updated successfully, but these errors were encountered: