Mac68k Forums

Home


Welcome, Guest
Guest Settings
Help

Mac68k Forums » Development » Software Hacking

Thread: Serial Disk Driver


Reply to this Thread Reply to this Thread Search Forum Search Forum Back to Thread List Back to Thread List

Permlink Replies: 72 - Pages: 5 [ 1 2 3 4 5 | Next ] - Last Post: Oct 29, 2013 12:19 PM Last Post By: bbraun
bbraun


Posts: 493
Registered: 7/25/12
Serial Disk Driver
Posted: Oct 13, 2012 11:24 PM
Click to report abuse...   Click to reply to this thread Reply
I was talking with techknight over on 68kmla about this, and the thread kind of got out of hand, so maybe things can be a little more focused here until things are ready for the wider audience. The goal is to use this driver with an embedded device that provides serial <-> SD card capabilities.
Serial is slower than most other interfaces, but it's ubiquitous, and conceivably works all the way back to the 512K, which doesn't have many other expansion options.

I've got the beginnings of a driver that will mount a disk image over the serial port.

serialdisk0.2.sit.hqx The MacOS driver. Throw the Extension in your Extensions folder.
serialserver0.2.tar.gz The POSIXy server software. Tested on SnowLeopard.

The protocol is pretty simple. The server is entirely passive. It will only send data in response to a request.
The request packet is in the form of:
1 byte command
1 byte imageid
4 bytes big endian offset
4 bytes big endian length
n bytes body

Responses are in the form of:
1 byte status
1 byte imageid
4 bytes big endian length
n bytes body

imageid is currently an unused field, but eventually intended to support multiple simultaneous images.
lengths and offsets are in bytes.

Valid commands are currently:
1 - Get Image Size. The offset and length of the request are currently ignored. The response length field indicates the size of the image (or whatever medium is being served up). This becomes the size of the "disk" to Mac OS.
2 - Read. The offset specifies the byte offset into the storage medium to begin the read operation, and length indicates the number of bytes to read. The response length should be the same as the request length. The response body contains the bytes read.
3 - Write. The offset specifies the byte offset into the storage medium to begin the write operation, and the length indicates the number of bytes to write. The request body contains the bytes to write. The response status is all that is observed by the driver. Response length is ignored.

The driver currently expects the server to be reachable on boot, and makes no attempts to retry if it isn't. I intend to add a polling mechanism to retry periodically, similar to the floppy disk polling for insertion status.
Ejecting the disk from Mac OS shows an error -17, but ejects successfully anyway. This is because the driver's control function is always returning error. Eventually, it should handle the eject control call and notify the server that the image has been ejected. I will probably have the server just exit on eject, and then the driver on the Mac OS side will poll to see if the server becomes available again.

The driver is using the Mac OS Serial Driver for communication, using asynchronous chained calls. It also passes the read/write requests straight through to the serial protocol. So if the OS issues a read for 100KB, the driver will issue that request to the server (I've observed a 100KB request size during app launches). Using the Serial Driver will make the driver play nicely with other apps that wish to use the serial port (it will report busy instead of clobbering each other), and the asynchronous nature of the calls will have better overall system responsiveness while serial operations are in flight.

Currently, 57600 baud is the fixed speed for both sides. The Mac OS Serial Driver doesn't support >57.6kbps. Macs equipped with a DMA capable serial port (AV macs, and later powermacs), have a special driver that supports additional driver calls to set 115kbps and 230kbps. This code doesn't support those calls at this time.
It may be possible to get additional speed out of the serial port by manually configuring the SCC chip, or by providing an external clock.
bbraun


Posts: 493
Registered: 7/25/12
Re: Serial Disk Driver
Posted: Oct 14, 2012 12:09 PM   in response to: bbraun in response to: bbraun
Click to report abuse...   Click to reply to this thread Reply
After doing some more reading about the mac's serial port, to get >57.6kbps requires manually banging on the serial controller directly.

Localtalk doesn't use the serial driver and manipulates the serial hardware its self. It enables interrupts, but when a packet is received, it disables all interrupts for the system, and polls in the remainder of the packet. This means it can have interrupts disabled for extended periods of time. In order to minimized dropped frames from the other serial port, the localtalk driver polls the modem serial interface (port A) and buffers the received traffic. Similarly, the floppy driver does this too, since it disables interrupts when working with the floppy drive.
For the serial driver to generate an interrupt on every byte received, >57.6kbps would probably generate too many interrupts to reasonably handle and maintain system performance.

The localtalk approach would work for the serial disk driver would work as well, if we limited communication to 512byte chunks, rather than passing through the request the disk driver received. To do this effectively, the serial disk driver would probably need to register an interrupt handler for the serial port's interrupt. It could send using entirely polled IO, return asynchronously to the caller, wait for an interrupt, and assume when we receive the interrupt, all required data will be present. It'd then disable interrupts, and poll in the response header, which would contain the remaining bytes to be read, which could be polled in.
However, given the largish sizes being read/written, I'm concerned about the impact on system performance with this approach.
bbraun


Posts: 493
Registered: 7/25/12
Re: Serial Disk Driver
Posted: Oct 14, 2012 4:18 PM   in response to: bbraun in response to: bbraun
Click to report abuse...   Click to reply to this thread Reply
serialdisk0.3.sit.hqx
serialserver0.3.tar.gz

Updated version is as before, but periodically checks (every couple seconds, up to 7 seconds) to see if there is a serial server connected. This means the serial server doesn't have to be there on boot, you can fire it up at any time after boot, and the driver will check for it. It checks for an attached server by periodically sending the "get image size" command. If no response is received within about 7 seconds, it will timeout waiting for the response, then it will wait about 1 second, and reissue the request.

This also adds a new command:
command 4 - Eject
When the disk is ejected on the Mac, the driver will send command code 3 to the server. Right now, the server immediately exits (without sending a response). The exit is to prevent the driver's periodic checking for a disk from picking up the same disk and just reattaching after it has been ejected. The driver will continue to check for the server to reattach, and will (re)mount the disk if the server comes back.

At 56kbps, this isn't a terrible experience. It seems fine for accessing an archive to copy files to local storage. With a (prefetching?) cache in the driver, it could even be pretty reasonable. It could also be interesting to use this as a mechanism to populate a RAM disk, although with a proper cache, that'd be essentially the same thing.
The server could even store a block access list persistently, for the driver to pre-fetch commonly used blocks. For instance, mounting a disk always accesses the same blocks in the same order.

I'm concerned about the baud rate and reliability though. The Plus can't do >19.2kbps reliably at all, and if localtalk is in use, it falls off fast (4800 baud?). Localtalk use on the other serial port will pretty much kill reliability at >9600baud for almost all of the 030 and many of the 040 machines. Floppy use at those speeds will be detrimental as well. The IIci and IIsi using internal video have reliability problems at 56kbps due to the RBV architecture.
Right now, the serial protocol has no retry ability, and no CRC of the data. There is a lot of work to do in the reliability area.
bbraun


Posts: 493
Registered: 7/25/12
Re: Serial Disk Driver
Posted: Oct 15, 2012 12:16 PM   in response to: bbraun in response to: bbraun
Click to report abuse...   Click to reply to this thread Reply
serialdisk0.4.sit.hqx

This update attempts to handle server disconnection as gracefully as possible. Unfortunately, Mac OS prior to 9 doesn't have a notion of "the disk has gone away and is never coming back". The floppy disk driver has a notion of the disk isn't available, but it posts a non-cancellable dialog saying "Please insert disk <foo>". This driver doesn't support reconnection, so even if the server does come back (and has the same disk image, which it might not), this isn't what we want.
All serial operations have a timeout now, and I've made a best effort to handle all the error cases. If a command times out, or any error occurs, the driver assumes the server has gone away and immediately returns error for every subsequent operation.
Unfortunately, Mac OS handling of those errors is suboptimal. Finder seems to frequently ignore the returned error values, errors in the DskErr global variable, and the fact that I'm saying no bytes were successfully read/written. So, basically the driver shouldn't hang the system or otherwise do bad things, but that's about the best it can do, I think.
techknight

Posts: 110
Registered: 10/13/12
Re: Serial Disk Driver
Posted: Oct 15, 2012 7:53 PM   in response to: bbraun in response to: bbraun
Click to report abuse...   Click to reply to this thread Reply
Man. your moving alot faster than I. hehe.

Once i get this research and development project done for work (which has been months now). I will have free time again to get coding on the hardware.
landonf


Posts: 86
Registered: 7/23/12
Re: Serial Disk Driver
Posted: Oct 16, 2012 2:11 PM   in response to: bbraun in response to: bbraun
Click to report abuse...   Click to reply to this thread Reply
I've been trying to get this working with Basilisk II's support for executing a program to serve as the other end of the emulated system's serial port via stdin/stout. Unfortunately I think I'm running up against weird Basilisk bugs that I haven't been able to figure out; no matter what software I "attach" to Basilisk's serial port (including an immediate call to sleep()), when the MacOS driver writes to the serial port, the following byte sequence is read back in:

0x5e 0x41 0x5e 0x40 0x5e 0x40 ...

ASCII representation is ^A^@^@^@... ring a bell for anyone?

I tried attaching a dtrace probe to write(), but I'm not seeing anything write out that byte sequence (but I might be using dtrace incorrectly, which isn't too hard to do).

I would up adding some basic getopt_long() support, along with --stdio and --disk flags to the serialserver. If I can't figure out what Basilisk II is doing wrong, I'll just post the changes here for posterity once I get home.

(edited) Fixed escaping of the ASCII
bbraun


Posts: 493
Registered: 7/25/12
Re: Serial Disk Driver
Posted: Oct 16, 2012 2:31 PM   in response to: landonf in response to: landonf
Click to report abuse...   Click to reply to this thread Reply
That's odd, that sequence isn't immediately ringing any bells.
Have you tried using zterm or similar for basic testing of the serial functionality from the macos side?

And yeah, the serial server could use some extra attention. It was mainly intended as a testing and debugging tool, since the ultimate goal is a standalone server that talks to an SD card. But we can definitely add some additional functionality.
What do you envision the stdio option doing, using stdin for the serial port? Or using the buffered stdio FILE* routines to access the disk image and serial terminal access?
landonf


Posts: 86
Registered: 7/23/12
Re: Serial Disk Driver
Posted: Oct 16, 2012 4:46 PM   in response to: bbraun in response to: bbraun
Click to report abuse...   Click to reply to this thread Reply
bbraun wrote:
What do you envision the stdio option doing, using stdin for the serial port? Or using the buffered stdio FILE* routines to access the disk image and serial terminal access?

Just performing reading/writing on stdin/stdout instead of via device, for compatibility with Basilisk's serial-exec functionality.
landonf


Posts: 86
Registered: 7/23/12
Re: Serial Disk Driver
Posted: Oct 16, 2012 7:50 PM   in response to: bbraun in response to: bbraun
Click to report abuse...   Click to reply to this thread Reply
Here's a patch against the latest serialserver-0.3 code. I still haven't actually gotten this working with BasiliskII, but I'm pretty sure it's not related to any issues with the serialserver.
bbraun


Posts: 493
Registered: 7/25/12
Re: Serial Disk Driver
Posted: Oct 16, 2012 7:56 PM   in response to: landonf in response to: landonf
Click to report abuse...   Click to reply to this thread Reply
Awesome, thanks. Applied locally and will include it in the next tarball.
landonf


Posts: 86
Registered: 7/23/12
Re: Serial Disk Driver
Posted: Oct 16, 2012 8:51 PM   in response to: bbraun in response to: bbraun
Click to report abuse...   Click to reply to this thread Reply
Attachment basilisk.png (62.8 K)
So apparently I'm an idiot and I needed someone else to point out the obvious:

landonf wrote:
0x5e 0x41 0x5e 0x40 0x5e 0x40 ...
ASCII representation is A@@@

Guess what that is? 0x1, 0x0, 0x0, 0x0 ... in other words, the escaped message that was written to the tty. Gwynne Raskind had to point out that it looked like escaped control characters to me. I feel like a moron :)

Basilisk II wasn't setting the TTY to raw mode, so:

  • Data was being buffered, hence my never seeing it be received by the serialserver, and
  • Data being written was being echoed back, but with control codes escaped.

With the basilisk bug fixed, it almost works:



I had to modify serial_unix.c to configure the master tty for raw mode, and add the following configuration line to the Basilisk config:

seriala |/Users/landonf/Downloads/serialserver/serialserver --stdio --logfile /Users/landonf/Downloads/serialserver/logf --disk /Users/landonf/Downloads/serialserver/disk
landonf


Posts: 86
Registered: 7/23/12
Re: Serial Disk Driver
Posted: Oct 16, 2012 9:23 PM   in response to: landonf in response to: landonf
Click to report abuse...   Click to reply to this thread Reply
Attachment basilisk.png (61.9 K)
I went ahead and committed my small Basilisk II patches here: https://github.com/landonf/macemu

To build on ML:

git clone git://github.com/landonf/macemu.git
cd macemu/BasiliskII/src/Unix
sh autogen.sh
env CFLAGS='-m32' LDFLAGS='-m32' CPPFLAGS='-m32' ./configure
make
./BasiliskII

It can also be helpful to turn on the DEBUG and MONITOR flags in serial_unix.c

I wasn't able to format the drive over the serial connection, but once formatted, the drive does mount:



Additionally, Basilisk II occasionally locks up in pty_make_controlling_tty() in the child process, causing the parent to lock in waitpid(). I haven't bothered to investigate the pty_make_controlling_tty() implementation; A quick work-around is to kill the stuck forked BasiliskII process.
bbraun


Posts: 493
Registered: 7/25/12
Re: Serial Disk Driver
Posted: Oct 17, 2012 1:24 AM   in response to: landonf in response to: landonf
Click to report abuse...   Click to reply to this thread Reply
I figured out the problem you had with the formatting.
First problem is with the server: I'm doing something stupid with returning the length.
Second part of the problem is with the driver. I'm not handling the returned value correctly (which probably lead me to do the stupid thing in the server, sigh). And I'm not handling a control call and a status call in the driver correctly for formatting the disk.
I've implemented those calls, and formatting works, but it's not quite right. The Formatting dialog is displaying the disk size in KB, followed by some garbage. I'd like to figure out what the garbage is.

But in any case, it's probably best not to format over serial if you can help it. Formatting is writing about 9KB, then reading it back, then writing the 9KB out again. For the entire contents of the disk. It's not that fun over serial.

Anyway, I'll post something tomorrow for formatting, although I'm not sure I'll have figured out the garbage display by then.
bbraun


Posts: 493
Registered: 7/25/12
Re: Serial Disk Driver
Posted: Oct 17, 2012 3:07 PM   in response to: bbraun in response to: bbraun
Click to report abuse...   Click to reply to this thread Reply
serialdisk0.5.sit.hqx
serialserver0.5.tar.gz

This time, it's a Control Panel! Please remember to remove the old extension before installing this. The driver and the INIT to load it live in the control panel, and bad things happen if you have both.

This fixes the format problem (previous versions would show the disk initialization dialog with a 0 sized disk), both in the server and in the driver. It also shouldn't do as much IO when doing the formatting, so it's not as bad as I mentioned in my post last night.
The Control Panel shows the mount status of the disk, along with connection status with the server. If this shows mounted/disconnected, that's bad and you should eject the disk.
The CP also lets you change the baud rate. You get the choices of 9600, 19200, 38400, and 57600. To support this, the server also takes a --baud option. Make sure the two match.
The server also includes landonf's patch to take long options and specify the device, disk image, etc. so remember the usage will have changed. But, there is usage information now too!
The control panel's icon is reusing the HTTP Disk control panel icon. Sorry about that, in fact I reused a lot of stuff from the HTTP Disk project in this.
bbraun


Posts: 493
Registered: 7/25/12
Re: Serial Disk Driver
Posted: Oct 18, 2012 2:58 PM   in response to: bbraun in response to: bbraun
Click to report abuse...   Click to reply to this thread Reply
serialdisk0.6.sit.hqx

Minor change to the control panel. If a disk is mounted, it'll show the name of the volume.


Point your RSS reader here for a feed of the latest messages in all forums