lainchan archive - /λ/ - 19368
[ Return /
Go to bottom ]
File: 1476539677095.png (125.36 KB, 300x105, application-binary-interface.png)
How does one create freestanding programs?
Given the complexity of today's software, I think it is refreshing to go back to basics. Using only a given processor instruction set and OS system call interface combination, how far would it be able to go? If a program does not link to any library, not even the C standard library, how would it work?
Given the complexity of today's software, I think it is refreshing to go back to basics. Using only a given processor instruction set and OS system call interface combination, how far would it be able to go? If a program does not link to any library, not even the C standard library, how would it work?
Just gonna mention you can make "freestanding" programs with static linking.
>not even the C standard library
then it will be a PITA to even get output to the console.
However, you can do just about everything, but painfully.
While I share your view, I wouldn't ditch the C standard library, why do so?
Plus there are some libraries which are decent and even useful, at the end, just KISS.
Oh and you can link statically too.
then it will be a PITA to even get output to the console.
However, you can do just about everything, but painfully.
While I share your view, I wouldn't ditch the C standard library, why do so?
Plus there are some libraries which are decent and even useful, at the end, just KISS.
Oh and you can link statically too.
I also have this fantasy about making an entire userspace atop the linux kernel that doens't even rely on such a thing as glibc. The userspace would be a lisp implementation with a custom layered architecture and access to all the system calls provided by the kernel.
It would be a no-C-applications allowed userland, it would only allow for lisp programs, strictly in source form, with it's own architecture for library access so that higher level programs would not need to be too big, rather they'd be quite simple.
It's just a dream though
It would be a no-C-applications allowed userland, it would only allow for lisp programs, strictly in source form, with it's own architecture for library access so that higher level programs would not need to be too big, rather they'd be quite simple.
It's just a dream though
Just fine basically. You're allowing system calls so there's really no problem. You'll probably want to write your own wrappers around them and I think you'll have to do a bit of inline assembly to actually syscall. Other than that it's just a question of not using any library functions.
Freestanding without system calls is a much bigger problem. You're basically writing a one program OS.
>>19371
>then it will be a PITA to even get output to the console.
It's not that bad. You still have access to filehandle operations through syscalls.
Freestanding without system calls is a much bigger problem. You're basically writing a one program OS.
>>19371
>then it will be a PITA to even get output to the console.
It's not that bad. You still have access to filehandle operations through syscalls.
>>19370
It's true... I didn't think of that!
I'd still need a library to link against, but it would still generate a fully self-contained executable.
>>19371
>why do so?
One reason is curiosity. What complexities are these libraries hiding from me? What are the actual capabilities of the underlying operating system?
When I looked up the operating systems that are popular today, the number of system calls varied by the hundreds. There must be functionality that isn't being exposed by the standard library.
Also, I think the Linux system call interface is much better designed than the C libraries. There is no errno, for example.
>>19372
I have a similar fantasy. One of the reasons I created this thread was to help shape my ideas.
I envision a small virtual machine that relies on nothing but the underlying operating system and processor. It'd simply provide a symbolic interface to the native platform. Besides a strong module system and sophisticated introspection capabilities, nothing else needs be offered. High-level programs can do the rest.
For example, what if we wanted to dynamically load a shared object on Linux? We can simply use system calls to open the file and map it to executable memory in the virtual machine's address space. Now it's possible to find the addresses of the exported symbols and make use of them.
It's true... I didn't think of that!
I'd still need a library to link against, but it would still generate a fully self-contained executable.
>>19371
>why do so?
One reason is curiosity. What complexities are these libraries hiding from me? What are the actual capabilities of the underlying operating system?
When I looked up the operating systems that are popular today, the number of system calls varied by the hundreds. There must be functionality that isn't being exposed by the standard library.
Also, I think the Linux system call interface is much better designed than the C libraries. There is no errno, for example.
>>19372
I have a similar fantasy. One of the reasons I created this thread was to help shape my ideas.
I envision a small virtual machine that relies on nothing but the underlying operating system and processor. It'd simply provide a symbolic interface to the native platform. Besides a strong module system and sophisticated introspection capabilities, nothing else needs be offered. High-level programs can do the rest.
For example, what if we wanted to dynamically load a shared object on Linux? We can simply use system calls to open the file and map it to executable memory in the virtual machine's address space. Now it's possible to find the addresses of the exported symbols and make use of them.
>>19372
>I also have this fantasy about making an entire userspace atop the linux kernel that doens't even rely on such a thing as glibc. The userspace would be a lisp implementation with a custom layered architecture and access to all the system calls provided by the kernel.
The libc library is just C wrapper functions over OS sys calls. I think its time to get away from the tyranny of C and provide these wrapper libraries in other languages. So what would be involved in re-writing libc in Lisp so we can have liblisp?
btw, here is a systems programming tutorial for Ocaml that might give some ideas:
https://ocaml.github.io/ocamlunix/
>I also have this fantasy about making an entire userspace atop the linux kernel that doens't even rely on such a thing as glibc. The userspace would be a lisp implementation with a custom layered architecture and access to all the system calls provided by the kernel.
The libc library is just C wrapper functions over OS sys calls. I think its time to get away from the tyranny of C and provide these wrapper libraries in other languages. So what would be involved in re-writing libc in Lisp so we can have liblisp?
btw, here is a systems programming tutorial for Ocaml that might give some ideas:
https://ocaml.github.io/ocamlunix/
>>19392
>ocamlunix
That's a pretty awesome page!
To this day I have no clue what's the proper way to handle signals. There's this weird concept of asynchronous signals safety and how you can't call pretty much any function because they might be interrupted by yet another signal while you're handling the first one. In the tutorial the OCaml runtime handles it somehow and then passess the signal to fhe program at safe points.
What's the proper way to do this?
Also, how is one supposed to handle SIGSEGV signals after an mmap error? It seems returning from the handler results in undefined behavior or something.
>ocamlunix
That's a pretty awesome page!
To this day I have no clue what's the proper way to handle signals. There's this weird concept of asynchronous signals safety and how you can't call pretty much any function because they might be interrupted by yet another signal while you're handling the first one. In the tutorial the OCaml runtime handles it somehow and then passess the signal to fhe program at safe points.
What's the proper way to do this?
Also, how is one supposed to handle SIGSEGV signals after an mmap error? It seems returning from the handler results in undefined behavior or something.
File: 1476644460765.png (11.26 MB, 200x200, The.Linux.Programming.Interface.2010.pdf)
>>19413
Start reading ch3 page43 and it will explain (with a diagram) how sys calls work in Linux and how they are wrapped with libc
Start reading ch3 page43 and it will explain (with a diagram) how sys calls work in Linux and how they are wrapped with libc
>>19423
It'd be interesting to know what is and isn't a system call and how much functionality they support. Linux man pages go straight to the C function, they never talk about the syscalls themselves.
For example, in order to manage memory malloc can just use [s]brk or mmap; sleep is implemented in terms of alarms which also explains why it doesn't guarantee much. It's not always clear what's going on when using glibc. How many system calls are involved in Windows user interface libraries? Can someone use them to make their own Windows API?
It'd be interesting to know what is and isn't a system call and how much functionality they support. Linux man pages go straight to the C function, they never talk about the syscalls themselves.
For example, in order to manage memory malloc can just use [s]brk or mmap; sleep is implemented in terms of alarms which also explains why it doesn't guarantee much. It's not always clear what's going on when using glibc. How many system calls are involved in Windows user interface libraries? Can someone use them to make their own Windows API?
>>19421
This is awesome, thanks!
>>19459
I am currently reading Advanced UNIX programming. it's a bit dated (last edition in 2003), but it's more general than just linux, however it doesn't seem to be as extensive.
TLPI, even bering strongly focused on linux, also documents POSIX system calls, and the differences between "just linux" and POSIX, so it is good as a stepping stone even if you don't use linux
This is awesome, thanks!
>>19459
I am currently reading Advanced UNIX programming. it's a bit dated (last edition in 2003), but it's more general than just linux, however it doesn't seem to be as extensive.
TLPI, even bering strongly focused on linux, also documents POSIX system calls, and the differences between "just linux" and POSIX, so it is good as a stepping stone even if you don't use linux
So, systems programming general?
I'm having trouble understanding what's the difference between non-blocking and asynchronous I/O. Can someone shed some light?
I'm having trouble understanding what's the difference between non-blocking and asynchronous I/O. Can someone shed some light?
>>19629
I see now that it was too early to ask that question. Having read more about the subject, what I'm really confused about is a limitation of the Linux kernel.
I understand I/O as a pipe. In reality, this `pipe` is really a kernel buffer that sits between the application looking to read or write data and the actual data sources and sinks like file systems and networks.
So, receiving data fills that pipe while reading data drains it. Data sources actually receive the data; network card receives packets, hard disk reads the requested blocks. The kernel then buffers the data. When the program issues a read system call, the kernel flushes the buffer.
Likewise, sending data drains the pipe while writing data fills it. When the program writes data, the kernel fills the buffer. Data sinks flush it by actually sending the data; network card sends packets, hard disk writes the requested blocks.
Is this an accurate understanding?
Anyway, non-blocking I/O seems to be all about when you can drain or fill the pipe while asynchronous I/O seems to be all about when the pipe has been successfully drained or filled.
It all seems quite convoluted to me. From what I've read, it seems Linux's epoll system call only supports non-blocking I/O. So, you can only be notified of when the pipe's full of data you can read or when the pipe's empty enough to write without blocking. You can't tell the kernel to read or write the data, forget about it and get notified later when the the operation has completed.
Because of this, you can't use epoll and friends on regular files. They're always in a `ready` state; just being able to open them at all implies readiness. It only makes sense for things like network sockets which can be opened but will only perform I/O at a later time. Also, even though you get asynchronously notified of when a file descriptor is ready, actually reading or writing to it still blocks, doesn't it?
Apparently, BSD's Kqueue supports these I/O operation completion events. Also, of all things, Windows is the OS that features I/O completion ports. Linux apparently only has legacy POSIX asynchronous I/O APIs, implemented using threads, even.
I see now that it was too early to ask that question. Having read more about the subject, what I'm really confused about is a limitation of the Linux kernel.
I understand I/O as a pipe. In reality, this `pipe` is really a kernel buffer that sits between the application looking to read or write data and the actual data sources and sinks like file systems and networks.
So, receiving data fills that pipe while reading data drains it. Data sources actually receive the data; network card receives packets, hard disk reads the requested blocks. The kernel then buffers the data. When the program issues a read system call, the kernel flushes the buffer.
Likewise, sending data drains the pipe while writing data fills it. When the program writes data, the kernel fills the buffer. Data sinks flush it by actually sending the data; network card sends packets, hard disk writes the requested blocks.
Is this an accurate understanding?
Anyway, non-blocking I/O seems to be all about when you can drain or fill the pipe while asynchronous I/O seems to be all about when the pipe has been successfully drained or filled.
It all seems quite convoluted to me. From what I've read, it seems Linux's epoll system call only supports non-blocking I/O. So, you can only be notified of when the pipe's full of data you can read or when the pipe's empty enough to write without blocking. You can't tell the kernel to read or write the data, forget about it and get notified later when the the operation has completed.
Because of this, you can't use epoll and friends on regular files. They're always in a `ready` state; just being able to open them at all implies readiness. It only makes sense for things like network sockets which can be opened but will only perform I/O at a later time. Also, even though you get asynchronously notified of when a file descriptor is ready, actually reading or writing to it still blocks, doesn't it?
Apparently, BSD's Kqueue supports these I/O operation completion events. Also, of all things, Windows is the OS that features I/O completion ports. Linux apparently only has legacy POSIX asynchronous I/O APIs, implemented using threads, even.
>>19634
I was going to answet your earlier question with what I can remember about *some* devices.
IIRC some devices can return a sort of "try again later" flag, kind of like when the buffer is full, or it has no data, like you just said.
For example, in reading or writing to a network socket, the TCP layer below may have "blocked" communications momentarily due to congestion (IP, at least, has a flag for this). Either of the communicating programs may try to use the channel asynchronouslly. You might want to write to it and it might immediately return a flag telling you: "can't do that right now, try again later" without hanging in the system call before it is possible to do so.
That would be non-blocking.
Asynchronous would be for example with a file where the resource usually need not be "ready" for writing, you need just dump into a buffer.
>Is this an accurate understanding?
one sinks and the other fills, yeah, that's pretty much it. IPC sex.
>while asynchronous I/O seems to be all about when the pipe has been successfully drained or filled.
non-blocking is asynchronous in nature, in the sense that it doesn't hang your process, but it also doesn't mean you can assume the operation was carried
> Also, even though you get asynchronously notified of when a file descriptor is ready, actually reading or writing to it still blocks, doesn't it?
I'm guessing this depends on the communications channel, pipes probably block, but other methods return this flag, that's non-blocking.
I was going to answet your earlier question with what I can remember about *some* devices.
IIRC some devices can return a sort of "try again later" flag, kind of like when the buffer is full, or it has no data, like you just said.
For example, in reading or writing to a network socket, the TCP layer below may have "blocked" communications momentarily due to congestion (IP, at least, has a flag for this). Either of the communicating programs may try to use the channel asynchronouslly. You might want to write to it and it might immediately return a flag telling you: "can't do that right now, try again later" without hanging in the system call before it is possible to do so.
That would be non-blocking.
Asynchronous would be for example with a file where the resource usually need not be "ready" for writing, you need just dump into a buffer.
>Is this an accurate understanding?
one sinks and the other fills, yeah, that's pretty much it. IPC sex.
>while asynchronous I/O seems to be all about when the pipe has been successfully drained or filled.
non-blocking is asynchronous in nature, in the sense that it doesn't hang your process, but it also doesn't mean you can assume the operation was carried
> Also, even though you get asynchronously notified of when a file descriptor is ready, actually reading or writing to it still blocks, doesn't it?
I'm guessing this depends on the communications channel, pipes probably block, but other methods return this flag, that's non-blocking.
File: 1477423536113.png (299.04 KB, 200x150, picture_11.png)
