Make serial port debugging more useful

[nelgin]

When I turn on debugging for the serial port, I just get scrolling pages of
Serial Read ACIA and Write ACIA lines.

What would be more useful is if I could see the characters coming in and going out the serial port, printable characters, ascii and hex values. I'm trying to debug an issue where I'm losing characters and it's hard to tell at what point. Seeing the values would help me compare to what it's supposed to have received and printed and can also match up with tcpdump/wireshark captures.

[chrisn]

I could add something, yes. I'd want to check that the characters read by the ReadFile call actually end up being read by the emulated Beeb. I'm looking at the threading code and wondering if it's doing the right thing...

[nelgin]

With a higher speed modem, I'm dropping characters. I couldn't tell where it's happening. I have a bit of a strange setup but I think characters are being dropped.

I run a bulletin board which only has tcp/ip connectivity and sits on a VPS.
Under my desk is a linux box which has USB modems attached.
My windows box also has a USB modem and they're all attached to a local voip ATA which connect to an asterisk PBX also on a VPS.

So user dials in connects to my linux box which then answers the modem and opens a tcp/ip session to my BBS.

I've had the software author add debugging code into the program that handles the modem and I can see all the characters arriving from the remote side so somewhere between the modem and BeebEM the characters are getting dropped.

I tried with the Telstar glasstty terminal and that works fine so I'm pretty sure there's some loss between the modem and BeebEm.

Sorry to be a little verbose in a ticket but I thought a full explanation of the situation would be useful. If you need me to test anything or you want me to try some code with extra debugging in, let me know.

[chrisn]

That's quite ok, and definitely an interesting setup! If you're OK compiling BeebEm from source, I'll make a branch to add some debugging code.

[nelgin]

No problem. I might need some help to get it checked out using VS2019. I'm more of a linux person. I don't usually do point & click compiling :)

[chrisn]

I've just pushed a serial-debug branch that logs the data received by the serial port in BeebEm to the Debugger window. You don't need to enable any of the Trace options, it will log automatically.

It doesn't show data sent from BeebEm, though. Do you need that too?

[nelgin]

Nope, I don't need to so data from BeebEm to the modem.
Anyway, immediate results. beebem-debug.jpg shows that everything is received correctly by the modem. Incidentally, the viewdata specifications states something like if the rest of the line is just spaces then don't print them, just skip to the next line. That's something I need to implement in the server, just in case someone tries to ding me on it.

Anyway beebem-display.jpg shows that not everything received is being displayed.

I did notice a number of compile warnings, these two in serial.cpp

1>C:\Users\nigel\Source\Repos\beebem-windows\Src\serial.cpp(710,20): warning C4244: 'argument': conversion from 'unsigned int' to 'unsigned char', possible loss of data
1>C:\Users\nigel\Source\Repos\beebem-windows\Src\serial.cpp(848,20): warning C4244: 'argument': conversion from 'unsigned int' to 'unsigned char', possible loss of data
1>C:\Users\nigel\Source\Repos\beebem-windows\Src\serial.cpp(1118,15): warning C4834: discarding return value of function with 'nodiscard' attribute

Reminder, I'm compiling on VS2019 which may cause different errors not seen on earlier versions.

[chrisn]

The warnings are OK, but I've pushed a commit to fix them. I don't see the jpg images though...

[nelgin]

That's because I didn't attach them :) sigh

[chrisn]

Thanks! So if everything seems to be received, but not displayed, the problem is somewhere between BeebEm reading from the serial port and it presenting the data to the emulated Beeb. I need to find a way to reproduce the problem here, to track down what's going wrong.

Out of interest, what baud rates are you using?

[nelgin]

I'm using 9600 with Commstar. The modems negotiate 24400 and above usually. It works fine with the Telstar client so I wonder if it's a buffering issue. If you don't have a modem it'll be hard to emulate. I tried to use Telstar dialin but that's pegged at 1200 baud and I don't know any other dialup viewdata boards. If you do have a modem then my board has a local UK number. +44 33 0117 1548

Of course the problem could be in a number of places. I'm using commstar so it could be an issue with buffering between the modem line and commstar, or it could be between the rom and the display...hard to tell. I'm happy to run any debug code. If you just want to push it to the serial branch.

[chrisn]

I've set up com0com with BeebEm connected to one virtual serial port (COM5) and PuTTY connected to the other (COM6), just to get started. Instead of using PuTTY to send characters to BeebEm I could write a program that sends a steady stream of characters at 9600 baud and see what happens.

[nelgin]

Keep in mind that side that is sending data is doing it via a tcp/ip link until it gets to the modem, then it's getting pushed out as fast as it can. The modems may connect at 24400 or higher but the BBC can only receive at 9600 which is making me think something is getting lost in the buffer somewhere. I did just try Commstar II and same result.

[nelgin]

Any progress with this issue? Is there anything else you'd like me to try or details I can provide?

[chrisn]

I haven't really looked into this in more detail yet. In #116 (comment) you mentioned handshaking - what form of handshaking are you using?

[nelgin]

Looking back over what I tried, it looked like it was getting all the data into beebem but just not from there to the screen, so maybe it’s not the handshaking anyway, but I believe it’s probably xon/xoff. I’m not sure the BBC did well with hardware handshaking.

[nelgin]

It's been a couple of weeks. Any further development in this direction?

[chrisn]

Sorry, I haven't had a lot of time recently. I don't think we've found out exactly where characters are being dropped yet. So I'm wondering whether to add more logging / debugging code somewhere, or just rewrite the serial port handling code.

[nelgin]

Well, we know that all the data is received from the modem but isn't making it to the screen. It's reaching the debug since we the characters, if I remember correctly, so something after there is dropping it. Not sure if I can run more tests or not, I'd have to go back and look at the debugging stuff you added and how to activate it.

[nelgin]

It looks like this still hasn't been addressed.

tcpser is showing this has been received.

2023-11-05 00:52:13:190624:DEBUG:Read 27 bytes from socket
2023-11-05 00:52:13:190624::RS->|0000|1b 42 45 6e 74 65 72 20 61 63 63 6f 75 6e 74 20|.BEnter account |
2023-11-05 00:52:13:190624::RS->|0010|70 61 73 73 77 6f 72 64 1e 0a 0a |password... |

Only this gets displayed:

So something is getting buffered and not displayed.

If I hit enter (or # since it's viewdata) then it does momentary display the missing characters before clearing the screen.

[chrisn]

I hope to have something for you to test soon.

[chrisn]

Here's a test build that includes some fixes from here. I haven't been able to test this myself. Please give it a try, I'll be interested to know if it improves things. Thanks!

[nelgin]

Initial testing looks good. Setting the speed to 9600 and connecting in viewdata mode, I didn't see any dropped characters. The first time I connected, no input was accepted. I did a CTRL-F12 break and restarted commstar and resumed the connection and it worked fine so that was a bit odd. I'll need to do more extensive testing but so far so good. Thank you.

[nelgin]

OK. I'm about to replicate the "no input" issue. It seems to happen on the first connection after starting beebem. Start beebem, load the commstar rom, C then # for Prestel mode. I to initilise and set speed to 9600/9600. Enter to quick back to the menu. then C for chat mode. ATDTendofthelinebbs.com:6502^M (Yes, you have to press CTRL M to enter) - it connects but won't accept input. I can see in tcpser that no text is being sent.

If I press F12 to break, commstar resets and I go through the initialization again and now input it accepted.

[chrisn]

I was able to get it to connect once, in Comms rather than Prestel mode, but haven't been able to repeat it. I'll check what happens when you first start BeebEm or press Break.

[chrisn]

Here is another update. I'm now able to connect to the BBS reliably and log in as guest.

[nelgin]

Confirmed. So far so good. Thanks for your work on this.

[chrisn]

Have you also tested with a modem on a serial port? I wonder if that is also working now.

[nelgin]

I have not tried an actual modem on the serial port yet.

Bad news, after a while, I see my BBS sending data to BeebEm, and it's going through tcpser but not making it to Commstar.

I see "Break Delay detected" in tcpser, not sure what causes it. Anything I type is also sent, I can see it in tcpser but not echoed. I tried a BREAK and CTRL BREAK and no dice. I have to close and reopen BeebEm to get it working again.

[chrisn]

I've been testing again, this time with both endoftheline and nightowlbbs.ddns.net:6400, and it seems to be working well. Which tcpser version are you using?