Implementing programmable invocations of GDB awatch rwatch & watch

Problem:
The OS xyz uses, Doug Lea implementation malloc dlmalloc.
A heap abstraction is built on top of dlMalloc.


An 8 byte allocation takes place from the heap, resulting in the following memory layout.


+-----------------------+
|DL-mchunk 8bytes
+-----------------------+<<---malloc returned address
|Usable 8bytes
+-----------------------+
|Heap Poison 0x5a5a5a5a <<---Corruption.
+-----------------------+
|Heap Integrity check
|struct
+-----------------------+


The allocated 8 bytes takes a ride of its life time, goes through Fs code, Buffercache code, sCSI code and then
in the interrupt callback path its seen that the heap poison (0x5a5a5a5a) is corrupted. We work on this corruption because our ass is on the stack of the PSOD.


Constraints:
-The OS is for some reason doesn't like you setting hardware breakpoints. Essentially gdb watch,rwatch and awatch
operations are undefined.
-Its a showstopper bug with very low turn around time. So you dont have all the time in the world to read all of the
code and checkin diffs. its already 7 working days with 4 heads working on it.
-Its a hiesen bug, attaching a debugger changes the timing somehow that the bug isn't reproducible under the debugger.



Solution:
---------
A standard solution would have been placing a breakpoint at malloc, and setting at gdb watchpoint at the poison
bytes to catch the culprit. But .... the great OS has problems that as soon as set the breakpoint and continue
code execution the debug server code itself asserts and crashes. Without setting the watchpoint and just letting the
code run, the bug is no reproducible.


So as seen the debugger here is apparently broken, along with the code. So if you have good understanding of debuggers, you'd know that all that the debugger does from the command prompt can be done programatically, within the being
debugged program, for example breakpoints can be set using good old int 3. So my best bet is to programatically set a watchpoint like. This would avoid the timing issue involved with setting a breakpoint and setting things by hand, and also the debugger is broken and is not allowing to set a watchpoint.

ptr = malloc(8);
watch((uint64_t)(char*)prt + 8, //watch the poison address
4, //4 bytes
WRITE_ACCESS); //for write access



So how to implement the watch(address,nr_bytes,accesstype) ? function.

implementing x64 watch,awatch,rwatch commands
---------------------------------------------
DR0 to DR3 allow you set 4 hardware breakpoints, i.e how awatch is implemented. (read wiki article in reference)

So algo for wathc(address,nr_bytes,accesstype) is
1. Load address into DR0 using ..


2. DR7 controls which of DR0-DR3 breakpoint are enabled.
Bit 0 of DR7 is set to 1, to enable breakpoint at address placed in DR0 in step 1. = 0x1
Bit 16-17 control access-type to memory. 01b is for WRITE_ACCESS 0001b
Bit 18-19 control number of bytes to be monitored 11b is for 4 bytes 0 1 0 1 b = 0xD

So the control code to enable our breakpoints to be placed in DR7 is 0x00000000000D0001
(all those 0's are required because x64 expanded the debug registers to 64bit, and all unused bits must be 0, else you'd get a GPF.)


3. Load control code into DR7 using.


C'code would look like.

Replacing int 1 with handler of int 3
------------------------------------

So I ran the code with my programatically placed mouse traps.

As soon as the culprit tries to writes my monitored bytes (poison bytes), "INT 1" is invoked. I stress the point because "INT 1" because I expected int 3 to be invoked. INT 1 will be or may not be implemented for interactive debugging for your OS. If yes then cool, the OS will wait/freeze to be broken into by the debugger as soon the write access take place. Just attach the debugger and get the backtrace as to which code path tried to eat up the 0x5a5a5a5a poisoned cheese.

But ... If INT1 is not implemented, you can still swiggle around that stuff, by copying the IDT entry for INT 3 into the entry for INT1. (I know I'm sometimes quite awesome with tips)


Even if int 3 is not implemented just write an ISR for int 1, with a loop, MAKE SURE INTERRUPTS ARE ENABLED while you spin in int 1, because the serial debugger cannot break in with the interrupts disabled ;)

.int1
label: nop
jmp label

Yes so we got the culprit on the stack :) Once again computer science saved the day !

Caveats:
--------
For the keen obeserver, who understand allocations and x64 its obvious that many OThER steps were required to nail the bug. But all cannot be explained in a blog. BUT the essense and CRUX of the solution is explained in this blog.
For example, it a race, so many allocations are successful, and so are free, you must not breakin when and valid access takes place when Heap_Free is called at instances which are not corrupt (its easy some more assembly).


Some bare-bones code snippets and references.
---------------------------------------------
static __inline void
load_dr7(uint64 dr7)
{
__asm __volatile("movq %0,%%dr7" : : "r" (dr7));
}


static __inline void
load_dr0(uint64 dr0)
{
__asm __volatile("movq %0,%%dr0" : : "r" (dr0));
}


corrpution_fn() {
char *ptr=NULL;

ptr=malloc(8);
if(!prt)
//do whatever
ASSERT(ptr);

load_dr0((uint64)(ptr+8)); <- Load address to be monitored
load_dr7((uint64)0x00000000000d0001); <- Load DR7 control code

// simualte corruption :)
//memset(ptr,0,12)
}





-http://en.wikipedia.org/wiki/Debug_register
-GNU GDB 6.5 gdb/i386-nat.c
-http://msdn.microsoft.com/en-us/magazine/dd252945.aspx
-Google for blog "Under The Hood - Matt Pietrek" - X64 Article Questions & Clarifications, and look at question asked by Nilesh Padribi - http://blogs.msdn.com/matt_pietrek/archive/2006/04/27/585218.aspx (keeps on changing)
-Intel manuals (though not required for this exercise)

Brent Welch's Talk "My code is better than yours" .. ..

I took up Advanced Storage systems at CMU. Yes the one that is taught by Garth and Greg. The class concluded with a talk by Brent Welch from Panasas. The talk was centered on his experiences around writing Quality good code. I am so ashamed to have slept through half of the class (because I was preparing for a demo to Rob Ross another cool guy from ANL late night). Friends say that I make it quite obvious when I sleep in class (head down and doze, the only saving grace is I do not snore). Hmm to the point, So Brent did talk about a lot of things which I probably did not register as I was asleep, but here are some of the bits and pieces that I registered and I felt were pretty good.

Brent made it quite obvious he was a serious programmer. I mean dead serious. I just could smell it from his talk. I think he is a dude programmer. I have met only a few dude programmers that are real dudes (YOUVRAAJ KELKAR, KIRAN JOSHI and BORISLAV MARINOV). I mean let’s face it everybody else is just about average tapping of theirs ways to glory polluting the already polluted universe of spaghetti code. Btw there is a difference in being a big mouth jibroni ass and talk about everything from Turing machine to WSDL, but a few actually do write code.

Here is what Brent said, which I quite believe in.

1. If you cannot set the tab to (4)spaces (or whatever the convention is for a project)in your code, he would think what kind of dumb ass you are. This is what I personally feel when some dumb ass tries to make a ruckus about HIS programming style and doesn't want to change it in the greater interest of the project.

2. This one is something very new very nice absolutely amazing.

Brent once asked an XYZ programmer who is fond of rewriting code (quite like me) - What is the time required to understand somebody else's code. The programmers answer was, "Time required to rewrite the code?" That is so true, given a smart programmers versified code it’s quite a task to understand his frame of mind that resulted in the code. Quite like an artist’s painting only he fully understands it.

3. Writing quality code is underestimated!

Again, so true every TOM DICK AND HARRY wants to code/pollute. Pollute Pollute Pollute and pollute more. Half of them are just not that passionate - they are just in for a good time. There are basically only 2 types of programmers ones that are passionate and the others that are not. There is not much grading criteria for you if you are not passionate. Hiring people who are not passionate can ruin a team as they bring down the morale of a great team down in more than 1 way. At times when I did have a chance to knock of people who are not passionate about the project - I personally did take delicate care to see that they are out!!!!

4. He believes that "His good is better than everybody else's code"

I buy his argument completely. I usually think the same that my piece of code is better than the rest of the shit floating in SVN/CVS land. How can 2 programmers live in harmony with such an attitude? The point is not that he/me find appreciating somebody else's code too condescending. What it means that the programmer has a lot a respect and responsibility for his code. He would take it personally if somebody passes a unfavorable comment about his code. I think this sense of responsibility and the urge to maintain once repute is what makes quality code deliverable. Its a standard you set for yourself increasing it every time when you learn the best practices of the trade.

Hmm from unfavorable comments I remember how I lambasted one of my colleagues for a frivolous comment he made. Back in old days I did work in Windows User land for Unitrends and this very cool friend of mine worked on Xiotech a kernel mode kick ass project (WTF we all know it was good). He said "Kya be tu kya chavanni ka code likhta user mode me ?" (What code worth 25 cents do you write in user land?). OK …………………… THATS IT what followed was a 1/2 hour tirade as to how dumb fcuck shit piece he, his life and his code was. The comment hit me bad, I know it when I feel it it’s that suffocating feeling that I get in my neck. He could not understand what just happened and was just dazed. ... I wasn't. If I get an opportunity again it will fcuk the shit out of him again for his comment. I mean nobody comments on what I do for a living literally. May be I have grown up now and I won't react the way i did back them, but i'm sure i would react in some way.

My rule is quite simple, your code my friend is the best …. just don't compare it with my code or comment on my code. If you are hell bent on a comparison I simply have 2 word for you "CODE IT !!!" or else talk to my hand. And for the ones who I feel are really better than me (I know its dumb to create a false illusion of being better when you know you are not) I make their praise pretty obvious !o!

P.S no one is a born dude. There is no short cut to being a dude. Except Nikola Tesla everybody follows the power law when acquiring skills.

Simplicity !! in Software Engineering.

I have always learnt more than what is taught at academic institutes. This is mostly due to the fact I analyse whats being taught in extreme detail. This excruciating analysis usually means that I do not complete the entire syllabus, but that's the way it is. My analysis usually stops when I have understood the very simple fact on which the academic conclusion/lesson was based. As an example I took a sentence from a software engineering class which states

"Software should be simple"

Is it end of the lesson? Has everything been learnt ? No I don't think so!!!

The key questions left to be answered are

1. Whats the definition of simple ?

2. How do you measure simplicity ?

3. What are the parameters to the measuring function ?

3. How do you apply the concept in real life ?

4. Are there any patterns associated with the application of the concept ? (Patterns, patterns, pattern how much do I love to identify them)

Hmm, now if I pose these questions to people who claim to understand "simplicity" I will get 100s and 100s of variations in answers. Which one is correct ?. Thus I tend to conclude that real knowledge is tested when there are no correct answers and right answers are based on circumstances in which the concepts are applied.

OK, enough of the ramble and scramble. Here is what I learnt today at Carnegie Mellon that is worth sharing. Its a very sweet and simple definition of "Simplicity" itself

Simplicity: Its the art of maximizing the amount of work not to be done to do the "thing right".

Now can we answer the questions about simplicity ? I am quite sure that the above definition puts the practice of keeping thing simple in the right perspective ;). The questions posed about simplicity are now well answered, and this is what is truly learnt because, this will eventually helps us in application of the concept.

Cheers,

Faraz.

P.S: I'm dumb at times. I risk stating the very obvious for the smart people. But, hey this surely helps other dumb people like me. Me being dumb helps me constantly evolve into a smarter being.

Food for thought (Courtesy Wikipedia:Simplicity):

"Simplicity means the achievement of maximum effect with minimum means." — Koichi Kawana, architect of botanical gardens

"Things should be made as simple as possible, but not simpler." — Albert Einstein (1879–1955) "Simple things should be simple. Complex things should be possible." — Alan Kay

"You can always recognize truth by its beauty and simplicity." — Richard Feynman (1918–1988) "

Our lives are frittered away by detail; simplify, simplify." — Henry David Thoreau (1817–1862) "Simplicity is the ultimate sophistication." — Leonardo da Vinci (1452–1519)

"If you can't describe it simply, you can't use it simply." — Anon

"Simplicity of character is the natural result of profound thought." — William Hazlitt