Ask HN: Do any books on OS Design cover TCP/IP stack implementations and algos

4 points by ZevsVultAveHera a day ago | 6 comments

This topic seems to be cheerfully absent from most popular books on Operating Systems. Both from the USA and Europe (at least among English language publications). Is it because TCP/IP stacks are that trivial or they are that arcane and scary?

I cannot even recall hearing mentions of any book covering that field in all may years of doing IT stuff and reading CS literature. If no typical Operating System design books cover them are they covered in any other kind of publications?

checker659 a day ago |

Does an old book count? If yes, I believe "TCP/IP Illustrated, Volume 2" is about the implementation.

Unless you're looking for kernel level implementation, you could take a look at userspace implementations. Google for "DPDK tcp ip" or something to that effect.

ZevsVultAveHera a day ago | root | parent |

The age of book is irrelevant to me. Thanks, would take a look at that publication. I am indeed looking mostly for publications that cover these stacks in context of kernels. But even theoretical approach seems to be not that common topic among authors.

082349872349872 a day ago | root | parent |

IIRC, Samuel J. Leffler, Marshall Kirk McKusick, Michael J. Karels, and John S. Quarterman (1988), The design and implementation of the 4.3BSD operating system, Addison-Wesley, ISBN 0201061961 has what you're looking for — might want to double check a TOC though because this memory is from the 20th century.

ZevsVultAveHera a day ago | root | parent |

If it is similar to newer variants of "The design and implementation [...] [Free]BSD Operating System" I'm afraid that publication is very lacking in regard of what I'm looking for and details of how TCP/IP stack of FreeBSD works itself. These are in my eyes more of "general (semi)quick reference" then analysis of what is going on and why, what was behind choices of algorithms/solutions of problems and what these problems were. But thank you for bringing to attention this publication, it may be useful one day to some readers, I'm sure.

toast0 a day ago | root | parent |

I was going to suggest the book you mentioned, but as you mentioned it's not an description of the design process, it's a description of the system as designed.

I don't think I've seen a holistic document that tries to explain the whys of everything. You'd really need to just go through the history of changes, and look at contemporaneous mailing lists, announcements and conference presentations.

If you go back far enough, most of the algorithms are pretty basic. Lots of full array scans, which wasn't horrible because the arrays were tiny. The algorithms come from what was convenient at the time it became a bottleneck; lots of hashing, and then you increase the size of the hashtable because there's too many collisions (although sometimes that gets silly --- it doesn't make a lot of sense to hash port numbers when your hash table size is 64k, and it makes no sense to have a hash table size larger than that, because that's the size of the input space). And sooner or later you need to deal with having large numbers of cores all working on the same tables, and how that interacts with locking, NUMA, etc.

If there's some choice really puzzling you, you can probably ask on the freebsd-net mailing list, and get a reasonable answer.

ZevsVultAveHera a day ago | root | parent |

Thank you for your detailed comment.

I was hoping that maybe there is some book covering things like the one you described in the longest paragraph of your reply. I'm personally of opinion that such literature would benefit humanity greatly or at least that part that is doing design and programming around parts where networking meets kernels (in general, I'm also interested in non-UNIX kernels) and their internals.

Of course doing research on codebase itself is a potential way to go but I know too well from my experience years ago that it often leads to scratching your head about some parts for weeks without clear or any answer. Good literature that presents practical problems and presents different attempts on solving them has IMHO a good track record of at least decreasing amount of confusion while analyzing such systems. It also "somewhat offloads" community (developers included) from "consulting burden". As sometimes it takes even years to get answer to some technical questions.