Data loss for USB CDC

[Forg_Bodo]

Hi,
I use the TMS Async for communication between an USB CDC and a windows PC. Basically the CDC streams a relatively large data amount from a sensor to the PC, approx. 400kB/s. Unfortuantely these are not equidistant distributed (time-wise), they come in chunks of about 40kB every 100ms. Each transfer of 40kB takes about 5ms.
The 40kB are splitted in packets with usually 4kB. Relatively often one of these packets is missing. It is more or less random, but I would say I lose every 20th packet.
The same code works eprfectly well for CDC devices with smaller data amounts, I have it running with a sensor which is transferring approx. the half data amount (but in the same time scheme).

I never get an overflow, or at least the overflow event is never triggered.
Also, I read the buffer always in time, ReadBufUsed was never >12kB and Buffers->ReadSize is set to 65536.
I also tried to split down the data into smaller packets with 256 byte only. The result was basically the same: I will lose a packet very fast.
Also, if I sniff the communication using wireshark (USBPCap) it gets clear that I will always lose an complete packet. It seems to me somehow that this packet even didn't fire an OnRXChar event.

What else can be the problem?

[brunofierens]

If you are sure that ReadBufUsed never reaches the limit, the loss cannot be in the app not reading out the big buffer fast enough.
Typically, there is a small hardware buffer that the component reads in its separate thread and moves the received bytes to its bigger buffer. Do you monitor the OnError event or OnRx80Full event on VaComm to see if this is triggered? Can it be there is code in your app that could make this VaComm thread unresponsive? Do you monitor CPU usage?

[Forg_Bodo]

Hi Bruno,
thanks for the answer. I did indeed monitor the OnError event, which is never triggered.
Now I implemented the OnRx80Full event and that has an interesting behaviour.
On EVERY incoming packet it fires. It seems that the OnChar event fires first, so that usually I would read out the buffer first, leaving the OnRx80Full pending.This would result in wrong readings if I call ReadBufFree() / ReadBufUsed() in the OnRx80Full event, so for a first test I removed the reading of the buffer in the OnRxChar event. Of course then I get identical results in both events. Now here comes the intersting part: It triggers even without getting close to 80% of buffer usage.

Here is a screenshot of the fired event and the results. It reads back 65351 free bytes and the OnRx80Full event is fired. To proof that it is fired from the same VaComm I did show you the sender, it comes from the same address. (In my code there can be multiple VaComms, tied to the struct DevConst[]. Nevertheless, the VaComms are created dynamically during run time and in this test we only have one).

Do you have any idea why the OnRx80Full event is fired, although the usage is far from 80%?
For your information: I raised the event priority to tpTimeCritical to see if that improves the performance, but so far no improvement. Therefore I don't think that something in my application can block the VaComm thread.

Do you think I should try a VaBuffer instance? Or does it only allow the preview but has no influence of the performance?

By the way, CPU usage rises during communication on one core up to 30%, I think that's pretty moderate.

[brunofierens]

OnRx80Full refers to the hardware buffer, not the software buffer VaComm internally has.
When it is triggered, it seems to indicate that at least something cannot keep up with the speed of transmission.
Have you considered using (hardware) flow control (CTS/DSR) to handle at hardware level there is no hardware buffer overflow?

[Forg_Bodo]

Hi Bruno,
using CTS/DSR is a bit complicated, since I use on the hardware side the USB CDC driver from Microchip Harmony, which doesn't support it.
Thanks for the info that OnRx80Full is triggered from the hardware, not from the SW buffer.
I've seen that the OnRx80Full event is triggered even on very small packets (<100 byte) when I send some control and configuration strings. I have the suspicion that this event somehow is triggered from the wrong hardware... If I have it correct in memory, the hardware buffer of a RS232 was 8 bytes. On Monday I will do a test with sending only 4, 5, 6, 7 and 8 bytes and see when it gets triggered.
If it gets triggered with small packets like these I can be sure that the event refers to the wrong hardware. Most likely this arises from the COM port emulation of the CDC. Nevertheless, I am sure that the hardware buffer of the USB controller must be much larger than a few hundred bytes. In that case I would ignore the OnRx80Full event.
I'll let you know how it turned out.
Regards,
Bodo

[Forg_Bodo]

Hi Bruno,
I did following test:
-Open the ComPort
-Send a single char to the USB CDC
-The CDC responds also with a single char

Although only a single character is received, the OnRx80Full event triggers immediately and reproducible.

So now I am a little bit out of ideas. I could of course implement some protocol with handshake to determine dataloss and try to resend, but it will be relatively complicated and with huge overhead.
Do you have any other ideas? Something else to test to get some hibnts from your side?

Best regards,
Bodo

[brunofierens]

That is very strange. I wonder if you find anything in the specs of the USB serial port device on internal buffering?
I guess that when you only send one char (i.e. one byte) and read this one char, you have no data loss?

[Forg_Bodo]

Hi Bruno,
no, in this case I have no data loss. Indeed I can send around 200kB/s without any data loss for long periods of time. Only if I increase the data rate up to 400kB/s it gets worse.

[brunofierens]

I can only guess that the internal thread in VaComm cannot always keep up with reading the data fast enough from the hardware buffer. If there is no flow control, that will obviously lead to hardware buffer overflow.