Fundamentals

A buffer overflow occurs when something very large is placed in a box far too small for it to fit. It's all gotta go somewhere. An example in code is as follows:

void func(void)
{
   int i;
   char buffer[256];   
                                  // *
   for(i=0;i<512;i++)                 
      buffer[i]='A';              // !
   
   return;
}

As you can see, our 'buffer' gets filled with 256 'A's, followed by 256 more that just don't fit. The rest of those 'A's have to go somewhere. And where they go depends on your operating system implementation and programming language, but if you don't have automatic bounds checking like Java, I guarantee you that those 'A's are going somewhere unfortunate. Here is a picture of a healthy 32-bit stack, in such an operating system as Windows 9x/NT running on an Intel platform. It looks like what it should look like at the point marked * in the code above.

STACK 
        ----------------
        Local Variables
ESP->   i
        Buffer
        ----------------
EBP->   Old Value of EBP
        ----------------
        Return Address
        ----------------

When the "func" procedure returns, it moves EBP back into ESP, and POP's the return address off the stack. When the above line of code marked '!' executes it overflows the buffer, writing 'A's over the old value of EBP and over the return address. By overwriting the return address, you can seriously alter the course of program flow. All you have to do is change the return address to point to a memory location of your choice, and the code you want to execute will be reached when this procedure decides to 'return'. If you stuff the buffer with code bytes, you can then reroute the EIP to them on the next RET, since the stack is considered executable memory in Windows 9x/NT on the Intel architecture. The lesson on basics is over. If you have written a buffer overflow exploit on other operating systems or have fully mastered these basic concepts, we will go into detail on how to recognize the buffer overflow condition in Windows and proceed to detail on exploitation.
What It Looks LikeWhen you see something like this,

you probably hit some kind of buffer overflow. Sure the error is somewhat generic looking, but look a little closer at some of those values... To get this to happen, I fed a string of 0x80 bytes into a popular conference package called 'Microsoft Netmeeting' through the address field of a 'speeddial' shortcut. EIP happens to be 0x80808080. Guess what? That's good! I found a stack overflow! Now all I have to do is craft my exploit string to have some fun code inside, and tweak four of those 0x80 bytes to point to my exploit string.
Note at this point that other types of errors will bring up similar dialog boxes, and that not all of them are buffer overflows. Some buffer overflows are easier to exploit than others as well. I will be going into the mechanics of stack overflows in Windows in this paper. Other types of overflows, such as heap overflows are exploitable, on Intel Win95/98/NT, but are beyond the scope of this paper by about 50 IQ points from the target audience.
Once you're pretty sure that you've found a buffer overflow, you need to decide what approach you're going to take, and find out what tools are available to you.