new stdio

New features and project activity between releases
Post Reply
alvin
Well known member
Posts: 1872
Joined: Mon Jul 16, 2007 7:39 pm

new stdio

Post by alvin »

I've placed the core stdio functions into cvs. The only things missing are getline/getdelim, which have been completed but I am unhappy with them so I will redo them later, and pipe which creates two fds, one to read a buffer and one to write to the same buffer. Pipes are normally used to communicate between a parent and child process but in z88dk I am thinking along the lines of communicating within the same program but in different bankswitched memory configurations. This can solve the problem of passing data between memory banks for some applications.

Getline / getdelim will require a better implementation of realloc to work well. getline / getdelim make calls to realloc a destination buffer one byte larger repeatedly so realloc should be trying to expand the existing buffer by one byte if a free block exists exactly adjacent to the existing block. Right now realloc does the naive thing by simply allocating a larger block and copying to the new large block. Realloc causes me a headache so I will probably put this one off for a while (which is an invitation to readers to take on the problem! :-)

I'm also unhappy with the implementation of file/stream-in/stdio_in_bkt which is the scanf converter %[]. It's a bit scatter-brained in implementation and can probably be done better. Again, another headache that I will look at later unless someone else feels up to tackling it in the meantime.

I elected to go with dynamic memory to create file structures rather than using a pool of some finite number of file blocks. The main reason was that I wanted device drivers and filter drivers to have the ability to request larger file structures to append any file-specific state data to a file structure. When drivers are called, they get a handle to their file structure which will include this additional state info associated with the specific file being operated on. Stdio's dynamic memory allocation is made from the "stdio_heap" which can be separate from the normal process heap (ie from which malloc occurs) or it can share the same heap as malloc. The idea is that stdio objects can be allocated from a specific memory configuration that may be active during stdio invocation and which may be different from the memory configuration in force when the stdio call is made.

I've also made sure there is a stdio_idle function, which can be customized, that is called whenever stdio blocks. This is an opportunity for the same to do a context switch or whatever when a specific process blocks on i/o. By default it's a simple ret instruction.

The new stdio uses a finite set of messages to perform operations. Those messages and associated register parameters are documented in stdio.def. Each open file is represented by a linked chain of file structures. A low level open() call will return a single file structure that terminates on a device driver. A dup or filter call creates another file structure that is linked to another fd. An fdopen call creates a FILE structure that is linked to another fd, etc. A stdio function, such as fgetc, takes as argument a FILE structure and passes a STDIO_MSG_GETC message to it. This message propagates down the stdio chain until something is able to satisfy the request. This could be a device driver or an intermediate buffer, eg.

Within the library, ix is normally used to point at the file structure being operated on. All file structures have a JP or CALL instruction at their head so that passing a message to the associated driver/fd/FILE is a simple matter of calling the file structure, ie, by calling the address in IX. So message passing in the code will always look like a register setup phase following by a "call l_jpix" or "jp (ix)" instruction. l_jpix is a library routine containing "jp (ix)". A driver, having been called, will have the address of its file structure following the initial CALL instruction on the stack.

Errors are typically reported at low levels by setting the carry flag. At the point where an error occurs, it is expected that ERRNO will be set appropriately. The calling function uses the carry flag to detect an error has occurred downstream and returns appropriately without altering ERRNO. The short routines in the error/ directory are useful for setting ERRNO or returning appropriate values in hl with carry set/reset upon detection of an error. Many of the entry points have one or more pop instructions preceeding them. This is to allow stack unwinding and a simple exit for calling routines.

This implementation of stdio is nearly a full one. As far as I know this is the first time this has been done for the z80 and further, the first time any form of stdio has been implemented in anything other than C. Please let me know if you think something is missing or see something that looks horribly wrong. Code inspectors are welcome :-)

I am currently writing the first tty drivers which will allow optional editable input and buffered key input. I've found this much more challenging than I thought it would be but I've got it mainly worked out in flowcharts so hopefully it won't be much longer before the first fully working target is ready for testing.
alvin
Well known member
Posts: 1872
Joined: Mon Jul 16, 2007 7:39 pm

Post by alvin »

Ah, yes, concerning previous discussion on this.. When opening files, a single character identifier is used to locate a specific device driver to pass the open request to. This driver identifier is placed in the first two chars of the filename string. If a driver identifier is not present, this open request is passed to a default device driver. The list of letter identifiers and device driver associations is held in a zero-terminated table. It is possible to dynamically grow and shrink this table which can be used to mount or unmount devices. There is no direct support for doing this as I think mounting and unmounting requires the notion of a global namespace which has not been implemented and if it were would sit on top of stdio.
stefano
Well known member
Posts: 1534
Joined: Mon Jul 16, 2007 7:39 pm

Post by stefano »

Quite a lot of new things.. impressive.
User avatar
dom
Well known member
Posts: 1194
Joined: Sun Jul 15, 2007 10:01 pm

Post by dom »

Very nice (ideas and code) I think an example implementation will make things a lot clearer - can I suggest that the "test" target might be a good initial target since it's pretty much a raw z80 and is unencumbered by a current io implementation as well as being shipped with z88dk so it's available for everyone.

Some hopefully constructive comments:

You might want to consider using a DEFVARS block to define the layout of the various structures - if you do the following:

Code: Select all

DEFVARS 0
{
    FILE_jp     ds.b  1
    FILE_next  ds.w 1
    FILE_flags  ds.b 1
    FILE_ungetc ds.b  1
    STDIO_SIZEOF_FILE
}
And use (ix+FILE_ungetc) then in theory that's easier to maintain than having a lot of wildcarded offsets - similarly the bit positions are probably better as defc constants.

I know most of the code was written before the assembler was capable of it, but can we use symbol entry points rather than the ASMDISP format - since the "user" implemented routines have multiple entry points it will be much less error prone.

If I understand things right, the underlying fd operations are all contained within a single file and will always be linked in if the device driver is being used? How will this affect the executable size?

Finally, are there any toolchain changes that would make life easier? - I'm guessing device inclusion can be handled via a #pragma.
alvin
Well known member
Posts: 1872
Joined: Mon Jul 16, 2007 7:39 pm

Post by alvin »

dom wrote:can I suggest that the "test" target might be a good initial target since it's pretty much a raw z80 and is unencumbered by a current io implementation as well as being shipped with z88dk so it's available for everyone.
I can do that but the only trouble with a raw test target is it's difficult to get any i/o going or to see results of i/o. I would like to incorporate sockets next so I was thinking of doing a test zx target since, apart from my being thoroughly familiar with it and the availability of excellent emulators with debuggers, there would be i/o available to see results and some real hw in Winston's spectranet to test socket incorporation into stdio. Spectranet is mainly a hw TCP/IP stack for which Winston has already written a C-like socket API. I see this as probably a lightweight task to incorporate spectranet into z88dk proper with some potentially intriguing results (ie IRC client, FTP client, etc).

I could turn test into a spectrum target, temporarily at least for testing purposes? Or perhaps it's better to leave test as a raw example for reference purposes and introduce another temporary zx target?
* You might want to consider using a DEFVARS block to define the layout of the various structures
* similarly the bit positions are probably better as defc constants.
* can we use symbol entry points rather than the ASMDISP format
Yeah this is probably the way to go. Once I get something working I will make one more pass through to make the necessary changes. I like the idea of having fgetc / fgetc_callee for C entry points, fgetc_asm for asm entry points and fgetc_lib for library entry points.
If I understand things right, the underlying fd operations are all contained within a single file and will always be linked in if the device driver is being used? How will this affect the executable size?
At the device driver there will be a jump table or equivalent containing a function address for each stdio message and this will draw in all the driver related code for all possible stdio messages. I don't think there is any way around this except for allowing users to remake drivers with the unused jump table entries substituted with an error routine. If we're going to put in pragmas to declare each driver, perhaps the pragmas could specify which jump table entries should be 'omiited' (as in substituted with a standard error routine). A compile would then necessarily involve first remaking device drivers and then linking that result to the main C program.

I'm not sure what toolchain changes should be made (or can be made!) yet -- this will probably become clearer as the test target fills out. For sure, printf and scanf can selectively omit specific % converters so detecting which is present and then automatically including only those needed (with the user having the option to add using pragmas?) will help with code size. It turns out that a lot of code is shared among the converters so omitting one may only save a few dozens bytes though, apart from the more unique ones (%[], float, eg). There are also two versions for each numerical scan converter : 16 bit only and 16/32 bit. The 16/32 bit version uses 32-bit arithmetic and is both slower and slightly larger. If both 16 bit and 32 bit versions are mixed you get both the 16 and 32 bit arithmetic code pulled in. It is probably better to choose all 16 bit or all 32 bit converters from a code size point of view. The IPv4 amd Mac-48 address converters both use the 16 bit arithmetic. The IPv4 input one, in particular, also draws in the 16-bit %i code. I suppose 32 bit versions of both these should also be written to help with the code size thing. I was thinking 32 bit versions should also do IPv6 and Mac-64 (?).
User avatar
dom
Well known member
Posts: 1194
Joined: Sun Jul 15, 2007 10:01 pm

Post by dom »

alvin wrote:
dom wrote:can I suggest that the "test" target might be a good initial target since it's pretty much a raw z80 and is unencumbered by a current io implementation as well as being shipped with z88dk so it's available for everyone.
I can do that but the only trouble with a raw test target is it's difficult to get any i/o going or to see results of i/o. I would like to incorporate sockets next so I was thinking of doing a test zx target since, apart from my being thoroughly familiar with it and the availability of excellent emulators with debuggers, there would be i/o available to see results and some real hw in Winston's spectranet to test socket incorporation into stdio. Spectranet is mainly a hw TCP/IP stack for which Winston has already written a C-like socket API. I see this as probably a lightweight task to incorporate spectranet into z88dk proper with some potentially intriguing results (ie IRC client, FTP client, etc).
My thinking was that we can easily add in hook (for want of a better word) codes in for the various low level ops - I got about half way through doing file handling last week which would give us something to validate the model against. Of course that makes it easy to add in a socket commands as well.

Having said that, if you do a spectrum test target since you're comfortable with that I'll extend the test target so there's more than one target - be quite interesting doing the socket api.

I'll port my ftp client when that's done: http://cvs.z88dk.org/cgi-bin/viewvc.cgi/zsock/ftp/ which should be pretty trivial since the network io is already abstracted out.

Jump tables are fairly easy to optimise and sort out with the appropriate pragma so that's that problem vanished.
alvin
Well known member
Posts: 1872
Joined: Mon Jul 16, 2007 7:39 pm

Post by alvin »

dom wrote:My thinking was that we can easily add in hook (for want of a better word) codes in for the various low level ops - I got about half way through doing file handling last week which would give us something to validate the model against. Of course that makes it easy to add in a socket commands as well.

Having said that, if you do a spectrum test target since you're comfortable with that I'll extend the test target so there's more than one target - be quite interesting doing the socket api.
Formal target independent testing would be very helpful. I wasn't planning to go that route myself since that is another job in itself but if you're on it, then that would be great :-)

I'd still like to do the test zx target to see some concrete results quickly and to get experience at writing the drivers. There may have to be more changes if the way things work makes it difficult to write efficient drivers.
I'll port my ftp client when that's done: http://cvs.z88dk.org/cgi-bin/viewvc.cgi/zsock/ftp/ which should be pretty trivial since the network io is already abstracted out.
Ha, yeah that would be great. Maybe we can get a +3 driver going and have the spectranet contraption ftping snapshots from WOS and saving them to disk.

With the tty driver I'm going to go whole hog. There's a separation between the code that manages tty aspects and a backend that needs to be customized for each target. It's looking like the target needs to supply three key input functions - READCHAR, PEEKCHAR and something like ICTRL. It will need tio maintain two perhaps small circular buffers, one for regular characters and one for out-of-band (OOB) characters (CTRL-C and CTRL-Z). When reading / peeking a char the input routine must supply OOB chars first if there are any. The ICTRL bit is for sending a reset/flush command and for indicating whether OOB chars should generate signals. In order for signals to be generated immediately they have to be dealt with at the low level input rather than at the tty driver. Implementation of this on the spectrum is fairly easy so I don't think the code footprint will be too bad. There can always be a simpler tty driver if it does get large.

The tty driver itself can ignore OOB chars (treat them as normal) or (default) separate normal and OOB chars into two streams. As per standard, reading the stream in normal mode when an OOB char is available returns an error with ERRNO indicating OOB data is there. The stream must then be flushed to empty OOB chars or switched to OOB mode to read the OOB data. The driver supports EDIT mode, which means line oriented input, or raw mode which means char at a time mode. non-blocking and echo on/off is supported.

The target specific output routines will likely include WRITECHAR (uninterpretted bytes), CTRLCHAR (including things like cursor right, left, etc) and something like ICTRL. The output routine needs to (optionally) maintain a concept of window size and take care of things like scrolling or wrapping.

I'm beginning to think the tty driver should manage input and output to a specific text window. It would still be possible to have several such windows active using the same code (just open() more from the same tty driver) but then there would need to be a concept of keyboard ownership which gets passed around as a token in order to control which window gets key input.

I am probably going to go with one low level target-specific key input / text output routine per compile. Each tty driver instance manages the input / ouput associated with a single text window. Switching the input token between windows will be the application responsibility (perhaps it reads a CTRL-TAB char or similar which causes ICTRL messages to be sent to one tty driver informing it it no longer has key input and to another tty driver that it now has input). So, any objections or better ideas?
Post Reply