[CS Dept logo]

Com Sci 230
Operating Systems

[back] Department of Computer Science
[] The University of Chicago

NACHOS Source Code

Last modified: Mon Feb 26 11:42:26 CST 2001


Guide to reading the NACHOS source

Thread Dispatching and Bootstrapping

Tracing execution sequence vs. following levels of abstraction

For well-written application programs, the best way to understand code is usually to read the object interfaces in the *.h files first. These interfaces, if they are well written and documented, will give you an overview of what each object is supposed to accomplish. Then, you look at the implementations of objects in the *.cc files, and see how the executable code supports each interface. The object definitions usually come in levels, where each level uses objects defined at lower levels in its own implementation. Each level introduces a new abstract functionality on which the next higher level is built.

Because of the trickery required to let an operating system control the activity of a number of simultaneous program threads, some aspects of OS code may be hard to follow in the usual preferred way described above. A crucial abstraction in a thread/process scheduler is the dispatching loop, which behaves like: 

While there are threads to run do
   Wait for at least one thread to be ready to run
   Run the next ready thread for a while
But, the implementation of this loop may be rather subtle, and may not be isolated in a single or even a small number of object definitions. Similarly, the startup and shutdown of the system may not follow the usual levels of abstraction.

In order to understand the main dispatching loop and the bootstrapping and shutdown operations in NACHOS, I found that I had to follow the execution sequence, rather than the levels of object implementation. I hope that for the higher-level services of the OS we will be able to use the more elegant style of reading code, which is usually much easier to follow when it works.

Why context switching is relatively expensive

In principle, a context switch when dispatching a new thread seems to be essentially the same thing as the initialization or restoration of processor state along with a function call or return. All of the memory associated directly with the CPU, PC, registers, etc., must be loaded consistently with the new code that is about to execute, instead of the old code that has just finished executing. But, there are ways to make function call/return more efficient, which do not apply to context switches between threads.

When a compiler produces code for a function call or return, it knows a lot about the code that comes just before and after. In principle, a given register only needs to be reloaded when it is used on both sides of the call or return. In order to keep the compile-time analysis of the code simple, most compilers either save and restore all of the registers used by the calling function, before the call and after the return, or they save and restore all of the functions used by the called function after the call and before the return. Either way, there may be many registers unaffected by a given call or return. There are also usually some registers, such as the Processor Status Word, that cannot be changed within a single program, so they never need to be saved/restored for a function call/return.

Threads, by contrast, tend to interleave in completely unpredictable ways, and they may be associated with different programs belonging to different users. So, a context switch between threads typically exchanges all of the memory associated with the CPU. On current machines that may be 10s or even 100s of words, involving about that many instructions executions. The context switch routines in the function SWITCH in switch.s appear to involve about 20-40 instructions, depending on the machine. The analysis of which registers are actually used would have to take place as the system was running, and would probably cost about as much, or even more than, just switching all the registers. From one point of view, the point of a memory cache is to take the use of some registers out of the direct control of the program, so that they can be loaded and saved as needed. The performance penalty of run-time analysis is avoided by in effect doing the analysis in fast special-purpose hardware.

Even supposing that we save/restore the same number of registers for function call/return as for context switching between threads, the function call/return has about a 2/1 advantage. Each function call and return corresponds logically to 2 context switches. But, since the called function always starts up at the moment of call, it has no saved register values to restore (except the PC). At the moment of return, the called function is finished, so there is no need to save its register values. But in a context switch, both threads are usually in the middle of their executions, so registers must be saved for the oldThread, and restored for the nextThread. Thus, a call-return pair probably costs less than a single context switch between threads. In principle, NACHOS could try to avoid restoring registers for a thread at its first dispatch, and avoid saving them at the last one, but this would only save one context switch (2 halves) out of typically 1000s or more, so it's probably not worth the extra code complexity and the cost of detecting the first dispatch and the last relenquishing of the CPU for each thread.

Function call/return also have the advantage of using a stack, while threads are stored on some sort of queue, which is usually a slightly more complex data structure. I suspect that this is a very minor point, though, compared to the number of register saves and restores.

Tracing the main dispatching loop

The innermost level of the NACHOS system, implemented in the preparation for the Thread project, supports interrupt handling and thread dispatching. The basic sequence of steps defining system operation is so simple that it is hard to see how the operation of the system gets rolling. The problem is that the main loop that drives the system is hidden in the pattern of calls to Run, Yield, and Sleep.

The heart of the operation of a thread system is a dispatching loop, that repeatedly takes the next thread on the ready queue (Scheduler::readyList) and runs it. In NACHOS, the dispatching loop is not implemented with the usual C++ iterative commands. Rather, it is implemented by a sequence of calls to Thread::Yield and Thread::Sleep in thread.cc, each of which then executes the call scheduler->Run(nextThread) to Scheduler::Run in scheduler.cc. These are the only two calls to Run in the initial NACHOS code. I recommend strongly that you do not add any more calls to Scheduler::Run.

In normal circumstances, when there is another thread ready to run, Yield and Sleep have exactly the same impact on dispatching: they each call Run on the scheduler. To see the differences between them, it is best to put the two function definitions side by side on two different windows or sheets of paper.

Impact on the old thread:

Yield puts the old thread on the readyList with the call scheduler->ReadyToRun(this).

Sleep marks the old thread as BLOCKED, with the assignment status=BLOCKED. There is no particular provision in Sleep to make sure that the old thread is ever made ready to run again. Rather, the system function that calls Sleep is responsible for insuring that some future event (in a different thread) will provide a wakeup call by executing ReadyToRun on the old thread.

Behavior when no other thread is ready:

Yield lets the old thread continue running if there is no other thread ready. This seems to be equivalent to sleeping with an immediate wakeup call. Notice that it would not work for Yield to execute Sleep followed immediately by a wakeup call in the form scheduler->ReadyToRun(this), since the old thread would have to wake up first in order to issue its own wakeup call. A wakeup call on one thread must be executed by another thread that is already running. Another variation on Yield would be to replace the conditional sequence 

nextThread = scheduler->FindNextToRun();
if (nextThread != NULL) {
    scheduler->ReadyToRun(this);
    scheduler->Run(nextThread);
}
by the unconditional sequence 
scheduler->ReadyToRun(this);
nextThread = scheduler->FindNextToRun();
scheduler->Run(nextThread);
It seems to me that by placing the old thread on the readyList in the first command, we guarantee that nextThread!=Null in the second line. The initial NACHOS scheduler runs the readyList as a FIFO queue, so the old thread would be scheduled precisely if there was no other thread ready. But, with a variant scheduler, e.g. using priorities, the old thread might get control right back in some cases even when there is another thread ready to run. Query: which behavior is more useful? Is there a fundamental reason to force a new thread to begin, even when the current one would be chosen as top priority? Is the form of Yield in NACHOS chosen just to avoid a dummy context switch of a thread with itself? Does context switching, which is accomplished by the command SWITCH(oldThread,nextThread) in Run, fail when oldThread=nextThread? The alternate form of Yield proposed above appears to be functionally equivalent to 
ReadyToRun(this);
Sleep();
It is crucial that these two commands are executed in the given order, and with interrupts disabled.

When there is no other thread ready, Sleep simulates idle time by calling interrupt->Idle(). On a real machine, this would be some sort of do-nothing loop. The idling ends at the next interrupt, which may or may not be the one that causes a wakeup call to the old thread.

Disabling of interrupts:

Both Yield and Sleep must execute the main part of their code with interrupts disabled. Yield disables interrupts at the beginning, and restores them to their old status (which might be enabled or disabled) at the end. Sleep assumes that interrupts were already disabled by the function that called it. I think that the assumption is considered sensible for Sleep because the function calling Sleep must also do something to cause the eventual wakeup call, and it needs to disable interrupts to avoid losing control in between Sleep and setting up the wakeup call. It wouldn't work to enable interrupts and set up the wakeup call before Sleep, because the wakeup call might come before Sleep, at which time it would either have no impact, or else it would wakeup the thread from another Sleep executed during an interrupt.

I am still puzzled by the necessity for these two ways of bundling the operation of giving up the processor along with other behavior. It seems that neither Yield nor Sleep, in the present form of the code, can be implemented conveniently and efficiently in terms of the other (Jim Firby pointed this out in a note). I like to think that there is a more elegant organization with only one way of suspending the running thread, but if it were easy to invent, no doubt the creators of NACHOS would have thought of it. Perhaps we should view Run as the true way of suspending a thread, and perhaps my puzzlement is just the result of the names chosen for the functions. Run sounds more like an initiator than a suspendor, while Yield and Sleep sound like suspendors. Perhaps we should view Run as a transfer of thread control, and view Yield and Sleep as two different complex operations, both of which perform a transfer.

The Scheduler relies on each thread to execute Yield or Sleep at reasonable intervals. Many threads will voluntarily execute Sleep in a system call to perform I/O, which executes as part of the thread that calls it. In the initial NACHOS code for the Thread project, the only call to Sleep comes from Thread::Finish (terminate a thread) in thread.cc, which is not very interesting since it only applies to a thread when it dies. Yield calls are a bit trickier. There is one from the test function SimpleThread in threadtest.cc, but that has nothing to do with overall operation of the OS. The only other call to Yield comes from Interrupt::OneTick in interrupt.cc. OneTick implements a tick of the universe's clock in the simulation of the MIPS machine. The location in code of this call is not realistic, since OneTick would not exist as software if NACHOS were running on a real MIPS machine.

The cause of the call to Yield from OneTick in the proper OS code is a call to Interrupt::YieldOnReturn from TimerInterruptHandler in system.cc. A code structure where the handler for timer interrupts causes the interrupted thread to yield is perfectly sensible and realistic. YieldOnReturn is defined in interrupt.cc to set a global flag that triggers the call to Yield from OneTick. I don't yet understand why this indirect trick is used, nor what it corresponds to in real unsimulated code. The internal documentation on this point doesn't make sense to me. The explanation in the NACHOS Road Map is more promising, but I haven't completely absorbed it yet. I also have a note from Jim Firby, who taught CS230 last year, with a sensible explanation. In any case, it appears that the tricky delaying of Yield has to do with the strange relation of the NACHOS OS running as a native process with the simulation of the MIPS machine. We may probably understand the code well enough by treating YieldOnReturn as an approximate synonym for Yield, in order to follow the main structure of the system.

To insure that no thread can monopolize the CPU while others are waiting, there is a (simulated) machine clock, called timer, that issues interrupts at regular intervals. As long as no thread is allowed to disable interrupts forever, no thread can monopolize the CPU. Since disabling interrupts is a privileged OS operation, a careful design can guarantee service to all waiting threads. A timer interrupt forces the running thread to execute the OS function TimerInterruptHandler in system.cc, which performs the call interrupt->YieldOnReturn() to Interrupt::YieldOnReturn in interrupt.cc discussed in the preceding paragraph.

Quick index to relevant code
Why not use an explict dispatch loop?

An explicit dispatch loop using a conventional iterative construct, such as while, would be much easier to read. Why does NACHOS use the much more subtle and delicate loop through function calls to Run? Notice that an explicit dispatch loop would not be part of any thread known to the thread scheduler, but it would still have an execution context of its own (registers, PC, stack, etc.). So, a context switch would be required to transfer control to the dispatcher, and another to dispatch a thread: two actual context switches for every logical thread dispatch. The NACHOS trick cuts that in half, which can be a big win for efficiency. In essence, an explicit dispatch loop behaves like a special thread that is never entered in the readyList, but dispatched directly every time another thread gives up control. Instead of directly switching from oldThread to nextThread, an explicit dispatch loop switches out of oldThread, and then into nextThread, as two separate context switches.

Of course, in computing there's always another way. We could program an explicit dispatch loop, making sure that it is very simple and uses only a few registers. Then, we could write special context-switching code SWITCHOUT to save the context of oldThread, while restoring only those registers that are used by the dispatch loop, and other special code SWITCHIN to save only those registers used by the dispatch loop, restoring the context of nextThread. The full context switch in SWITCH would no longer be used, and the sum of the work in SWITCHIN and SWITCHOUT would be just slightly greater than that in the current SWITCH. Coded carefully, this would probably be about as fast as the NACHOS dispatching method. But, we would probably have to write the dispatch loop in assembly language.

In some sense, a single thread is just a region of execution in which context switching is incremental---every register operation is essentially a tiny piece of context switching. By expressing the dispatch loop implicitly as NACHOS does, we let the compiler do the work of figuring out the tiny bit of context switching (register operations) involved in changing over between dispatching code and other code. By expressing the loop explicitly in assembly language, we are forced to do that work ourselves, and the clarity value of an explicit dispatch loop is certainly much less in assembly code than it would be in high-level code. Of course, some future innovation in compilers might let us have our cake and eat it too: express the dispatch loop explicitly in high-level code, but get the efficiency of the implicit method or the explicit assembly-code method. Given the currently available compilers for C++, the NACHOS method appears to involve the least assembly code to achieve its level of efficiency.

Starting and stopping the system

When you execute the nachos command from the UNIX shell, execution begins with the function main in main.cc. main calls Initialize, in system.cc, which initializes global data structures and starts the interrupt handler, thread scheduler, and machine timer.

Initialize sets up the interrupt handler, thread scheduler, and simulated timer with the normal C++ commands 

new Interrupt;
new Scheduler();
timer = new Timer(TimerInterruptHandler,0,randomYield);
which call the constructor functions for the classes Interrupt, Scheduler, and Timer, respectively. Then, Initialize uses the command currentThread=new Thread("main") to create the first formal thread known to the thread scheduler, indicates that it is already running with the command currentThread->setStatus(RUNNING), and makes it interruptible with the command interrupt->Enable(). Now, the remainder of the execution of Initialize, and the further execution of main after Initialize returns, are carried out as part of a thread named "main", which for now is the only thread in the system.

After Initialize, the remainder of main must start up whatever activity the OS is expected to control. The various segments of conditional code provide activities to test several of the functions of the OS as they are added. To run NACHOS as a real system, we would have the remainder of main start up one or more threads to monitor various I/O devices and start user jobs based on their input. E.g., to provide a login service similar to UNIX's, we would start threads all running a program to output a "login:" prompt, and interpret user commands as they are typed back. There would be one such thread for each potentially active terminal port.

The end of main is a bit subtle. After starting up whatever activity is appropriate for a given project, main gives the command currentThread->Finish(), which is a call to Thread::Finish in thread.cc. This marks the current "main" thread, under which the function main is executing, for deletion, by setting the global variable threadToBeDestroyed=currentThread, and then executes Sleep() to stop execution of the current thread. The next execution of the function Scheduler::Run, which will be called from Sleep, executes delete threadToBeDestroyed after switching to a new thread. It would be disastrous to for a thread to delete itself, leaving no running thread at all. Question for you: what happens when the last thread executes Finish?

After the "main" thread is deleted, other threads that were created by main will still execute to completion. Even though the thread executing main is gone, from the point of view of the nachos program itself, the function main is still active; otherwise, nachos would halt. The actual termination of nachos works like this:

Quick index to relevant code
The interleaving of function call/return with context switch

Every function call and return, except for the initial call to main, happens within some thread. Every function activation returns in the same thread in which it is called, although other threads may have executed in between the call and return. But, Run appears to be specifically designed to return in a different thread from the one that calls it. What is really going on?

Think of the implementation of function call as pushing an activation record on the recursion stack, and return as popping the stack. Each thread has its own recursion stack, and SWITCH makes the stack associated with nextThread into the one that is used for function call/return. So, a call to Run pushes an activation record on the stack in oldThread, but the next return to be executed, which is usually a return from a different activation of Run, pops the completely different stack in nextThread. When oldThread is selected again for execution, it will execute the return from its activation of Run, but the most recent call to Run will be in yet another thread, associated with yet another activation record. To make sure that you understand what happens in Run, figure out what values the local variables oldThread and nextThread have on the debugging line after the call to SWITCH: 

DEBUG('t', "Now in thread \"#s\"\n", currentThread->getName());
What would you see if you printed out oldThread->getName() and nextThread->getName() as well? Figure it out, and then try the experiment.

What is on the stack of nextThread when it first starts running again as the result of a call to Run? As long as nextThread has run before, it must have given up the CPU by executing either Sleep or Yield, each of which calls Run. That call to Run was immediately followed by a return (from a different activation of Run or some other function) in a different thread, so it is still sitting on the top of the stack for nextThread. The call to Run that is dispatching nextThread pops the activation record that was stacked by the call to Run that last stopped the execution of nextThread. From the point of view of the execution of nextThread, this looks just like a call to Run which has returned in the normal way. But, the return is associated with the activation of Run from the last call to it in nextThread, while the most recent call to Run was executed in oldThread, and its activation record is still on the saved-away stack for oldThread. The "main" thread that is bootstrapped by the function main starts out executing on the CPU, so it always has a dangling call to Run when it is on the readyList.

At first, I thought that the call to Run in oldThread, and the subsequent return from Run in nextThread, were co-ordinated in a delicate way, and that correctness would depend on each thread activated by Run having a dangling call to Run to return from. On further reflection, it seems that there is no fundamental reason why nextThread needs to start out executing code in the same function that oldThread ended in. It is really an accidental result of the fact that there is only one function that executes a context switch. Since Run is used by oldThread to pass the CPU on to nextThread, the functional structure of the code guarantees that the next time oldThread executes, it picks up in the latter part of Run, after SWITCH. But, a thread that has never run before doesn't need to have a dangling call to Run in it, as long as it accomplishes whatever tasks need to be done at the start of its first period running as currentThread.

In fact, the delicate part of Run is down in the machine-dependent assembly code for SWITCH. The return from SWITCH is executed in nextThread, since it has to come after the stack for nextThread has been restored as the actual function stack for the CPU. But, the saving of the PC for oldThread was done several instructions earlier. It is crucial that the saved PC is one that points to the code after the body of SWITCH, else some of that code will be executed twice. I don't know the various assembly languages well enough to be sure I'm reading correctly, but it appears to me that the PC that is saved for oldThread is the return address for the current activation of SWITCH, which points to the code in Run immediately following the call to SWITCH. If anyone can follow one of the versions of SWITCH in detail, pleast post your observations.

Starting a new thread with Thread::Fork

The function main bootstraps a single thread, but in order to do interesting things, we need to create as many more threads as are required to run users' programs. We need at least one thread per program in memory (on the "short-term queue", or "not blocked and not suspended", in the jargon of the text). Since we've got a nice thread-management system, we might as well allow users to program multiple threads in individual programs, too.

The function to start a new thread is Thread::Fork in thread.cc. The traditional name Fork expresses the programmer's view of the operation, which is used to make a single thread branch, or "fork", into two threads. From a programmer's point of view, it doesn't matter much which of the two future threads is thought of as the continuation of the old one, and which one is thought of as "new". Most systems, including NACHOS, treat the two very differently---one is just the old thread going on, and the other is brand new. In NACHOS (and in most other systems, I think), the old thread continues to execute, while the new one is entered on the readyList. This avoids one context switch, and it suits a style of programming in which one thread executes a loop to fork off a number of other threads of roughly equal priority.

From the point of view of NACHOS then, the action of Fork might better be thought of as create. Strangely, while the name of Run is chosen as if its job is to run a particular thread, but the system is better understood by thinking of it as transferring between two threads, the name of Fork is chosen as if its job is to make two threads run side by side, but the system is better understood by thinking of it as creating a single new thread.

The NACHOS Fork operation takes two arguments, func and arg. The behavior of the new thread created by Fork is to run the function pointed to by func on the argument arg. When func returns, the new thread kills itself by executing Thread::Finish. In principle, that behavior should be produced by executing the C++ commands 

interrupt->Enable;
(*func)(arg);
Finish();
as the first actions of the new thread.

Looking at the code of Fork, we see that all it does is create an initial context for the new thread using StackAllocate, and then put the new thread on the readyList with the command scheduler->ReadyToRun(this). Notice that the variable this refers, not to the currently executing thread, but to the new thread t for which t->Fork has been called. t will always be a brand new freshly allocated thread, as in the example call to Fork from SimpleThread in threadtest.cc.

Interrupts are disabled and then restored, because ReadyToRun is written under the assumption that interrupts are off, rather than disabling them itself. Notice that there is no need to prevent interrupts between StackAllocate and ReadyToRun, since the data manipulated by StackAllocate are perfectly safe as local data in the currently executing thread. It is generally most efficient to disable interrupts for the shortest time that is sufficient to guarantee correctness. Interrupts typically represent fairly high-priority tasks, and should be delayed as little as possible.

The delicate part of Fork is all handled within StackAllocate. StackAllocate must create an initial context for the new thread that will lead to the execution of (*func)(arg) and Finish. The C++ code in StackAllocate is machine-dependent (actually, compiler-dependent), because the structure that it creates must match the particular form of allocation records used by the C++ compiler on the machine running NACHOS (not on the simulated MIPS machine). Since the stack for the new thread is being created in memory, this initial context is in the form of data assigned into the array Thread::machineState, declared in thread.h as a private member of the Thread class to represent the context of a thread that is not currently running. When the new thread reaches the head of the readyList and starts running, a call to SWITCH from Run will make the data in this array part of the actual C++ recursion stack.

The delicate part of StackAllocate is the initialization of the PC for the new thread, by the assignment machineState[PCState]=(int)ThreadRoot. The C++ "cast" operation (int) takes the address of the first instruction in ThreadRoot and treats it as an integer, so it can be stored in the integer array machineState. What should be the code for ThreadRoot? At the moment, I don't understand why it cannot be the high-level code suggested above, but for some reason it is done instead in machine-dependent assembly code. The peculiar section at the end of thread.h that looks like 

extern "C" {
// Comments omitted
void ThreadRoot();
// Comments omitted
void SWITCH(Thread *oldThread, Thread *newThread);
}
states that the two functions ThreadRoot and SWITCH are defined elsewhere. Don't be confused by the juxtaposition of ThreadRoot and SWITCH in thread.h. The extern construct merely states that both of them are defined elsewhere, it does not call either of them, and it particularly does not establish that one will be executed after the other. The actual code for ThreadRoot is the machine-dependent assembly code following the label ThreadRoot or _ThreadRoot (there is one such label for each machine) in switch.s.

I think that something is missing from ThreadRoot. At the end of Run, just after the call to SWITCH and the subsequent DEBUG code, there is a short piece of conditional code to check whether the previous thread had executed Finish, and therefore its data structures should be reclaimed so that its memory may be reused in the future. This bit of code is the first thing executed when a thread gets the CPU, as long as that thread has run before. But, it appears that the first running period for a new thread created by Fork will not check whether the previous thread has Finished. In most cases, the Finished thread will remain as the value of threadToBeDestroyed until some future context switch leads to a thread that has been executed before, with a dangling activation of Run, so the deletion of unused data structures is merely delayed a bit. But, what if a newly Forked thread should take over the CPU from a Finished thread, and the newly forked thread should also Finish during its first running period? It appears to me that it will overwrite the value of threadToBeDestroyed, and the previous thread will never be deleted. This probably happens very infrequently, but I believe that the possibility is a bug in NACHOS. If you have another analysis of this problem, or think that NACHOS is correct in this respect, please post. As soon as we figure out how to compile and test NACHOS, we should determine by experiment whether there really is such a bug.

If I've understood the apparent bug correctly, then we can fix it by adding the thread deletion code from the end of Run to the function that is stored as machineState[StartupPCState] by StackAllocate. Currently, that function is called InterruptEnable, and its body is the single command interrupt->Enable(). If I'm right we should instead point machineState[StartupPCState] to a function, which we might call FirstCodeInThread, with the body 

if (threadToBeDestroyed != NULL) {
    delete threadToBeDestroyed;
    threadToBeDestroyed = NULL;
}
interrupt->Enable();
The conditional code before the enabling of interrupts is copied verbatim from the end Run. Notice that it is important to do the old thread deletion before enabling interrupts.
Quick index to relevant code