I see now that it was too early to ask that question. Having read more about the subject, what I'm really confused about is a limitation of the Linux kernel.
I understand I/O as a pipe. In reality, this `pipe` is really a kernel buffer that sits between the application looking to read or write data and the actual data sources and sinks like file systems and networks.
So, receiving data fills that pipe while reading data drains it. Data sources actually receive the data; network card receives packets, hard disk reads the requested blocks. The kernel then buffers the data. When the program issues a read system call, the kernel flushes the buffer.
Likewise, sending data drains the pipe while writing data fills it. When the program writes data, the kernel fills the buffer. Data sinks flush it by actually sending the data; network card sends packets, hard disk writes the requested blocks.
Is this an accurate understanding?
Anyway, non-blocking I/O seems to be all about when you can drain or fill the pipe while asynchronous I/O seems to be all about when the pipe has been successfully drained or filled.
It all seems quite convoluted to me. From what I've read, it seems Linux's epoll system call only supports non-blocking I/O. So, you can only be notified of when the pipe's full of data you can read or when the pipe's empty enough to write without blocking. You can't tell the kernel to read or write the data, forget about it and get notified later when the the operation has completed.
Because of this, you can't use epoll and friends on regular files. They're always in a `ready` state; just being able to open them at all implies readiness. It only makes sense for things like network sockets which can be opened but will only perform I/O at a later time. Also, even though you get asynchronously notified of when a file descriptor is ready, actually reading or writing to it still blocks, doesn't it?
Apparently, BSD's Kqueue supports these I/O operation completion events. Also, of all things, Windows is the OS that features I/O completion ports. Linux apparently only has legacy POSIX asynchronous I/O APIs, implemented using threads, even.