Wednesday, March 26, 2008

How to work with jsd (Javascript Debugger)

I was really determined to figure out how firebug and venkman works, so I can write a javascript walker or stepper for my firecruncher plugin. The further I investigate, the more I respect the authors and contributors of these two pieces of seriously written software.

I started with the UI, and then I headed for the core that interacts with the xpcom compionent jsd (Javascript Debugger).

It never seamed easy for me, even though there are some sparse documentation in xulplanet, but the only two examples that I was able to consult was
It really took me a while figuring out how jsd works, as both of the source above have thousands of lines of javascript, a lot of them has interactions with objects outside of the classes or services that are defined in the same file.

After a week's worth of night reading, prototyping and code porting, I finally got it. Well, most of it. ;-)

It turns out that using jsd isn't that hard at all if you know Javascript's event handing mechanism. Well, in jsd, the original designer called them hooks. There are hooks for almost everything from the creation and destruction of a script, errors, to function call and program counters (interrupt or execution). Just think of them as different events that will be fired during a debugger active session. All what we need to do is to write the appropriate handers, and hook back into the jsd.

I will describe a common life cycle of the usage of jsd below:

First, we acquire the @mozilla.org/js/jsd/debugger-service;1 service.
var jsd = Components.classes["@mozilla.org/js/jsd/debugger-service;1"].getService(jsdIDebuggerService);
jsd.on();
Ref: http://www.xulplanet.com/references/xpcomref/ifaces/jsdIDebuggerService.html
With this, we can turn on, off, pause and unpause the debugger, but how do we set a break point? Ha, we are jumping ahead of ourselves. We need a piece of script first, so we can set the break point on top of it. Here is how you track which script has been created and which one has been destroyed.
jsd.scriptHook =
{
onScriptCreated: function(script){/* xxx track the scripts xxx */},
onScriptDestroyed: function(script){/* xxx track the scripts xxx */}
};
You can also enumerate all of the existing scripts that are available before the debugger was turned on by:
jsd.enumerateScripts({enumerateScript: function(script)
{
/* xxx track the scripts xxx */
}});
Now, we have all of the scripts available for us to set the break points. ;-)
Ref: http://www.xulplanet.com/references/xpcomref/ifaces/jsdIScript.html

That's all good until we encounter an error while running the script that we have loaded into the debugger. What do we do then? Ha, the ErrorHook comes handy.
jsd.errorHook = { onError: function(message, fileName, lineNo, pos, flags, errnum, exc){ /* xxx handle the error here xxx */ }};
You can do similar things with debuggerHook and debugHook. So what happens when we encounter a break point? Of course, there is a break point hook as well ;-)
jsd.breakpointHook = { onExecute: function(frame, type, val){ /* xxx handle the break point xxx */} };
We can do something at a break point now, but how do we temporarily suspend the code execution while we are at a break point, and do something like step over? It took me some search to figure this out ;-) We use enterNestedEventLoop to pause, and exitNestedEventLoop to get out of the paused state.
jsd.enterNestedEventLoop({onNest: function() { /* xxx another hook here, so we know that we are in a paused state xxx*/ });
jsd.exitNestedEventLoop(); // Yeh.. we are happily continuing our course of execution.
OK, we now can get a script, handle the error, set the break point, handle the break point, pause at the break point and continue from the break point. So how do we do the stepping? This is done by two other interrupts: functionHook and interruptHook ;-) In the case of step over, we use interruptHook, in the case of step out and step in, we use functionHook. (Ref below is copied from firebug-service.js)
hookFunctions: function()
{
function functionHook(frame, type)
{
switch (type)
{
case TYPE_FUNCTION_CALL:
{
++hookFrameCount;

if (stepMode == STEP_OVER)
jsd.interruptHook = null;

break;
}
case TYPE_FUNCTION_RETURN:
{
--hookFrameCount;

if (hookFrameCount == 0)
fbs.stopStepping();
else if (stepMode == STEP_OVER)
{
if (hookFrameCount <= stepFrameCount)
fbs.hookInterrupts();
}
else if (stepMode == STEP_OUT)
{
if (hookFrameCount < stepFrameCount)
fbs.hookInterrupts();
}

break;
}
}
}

jsd.functionHook = { onCall: functionHook };
},

hookInterrupts: function()
{
function interruptHook(frame, type, rv)
{
// Sometimes the same line will have multiple interrupts, so check
// a unique id for the line and don't break until it changes
var frameLineId = hookFrameCount + frame.script.fileName + frame.line;
if (frameLineId != stepFrameLineId)
return fbs.onBreak(frame, type, rv);
else
return RETURN_CONTINUE;
}

jsd.interruptHook = { onExecute: interruptHook };
},
That's it!

Wednesday, March 12, 2008

Firebug's SourceText

Yesterday, I went through the source file loading between Firebug and Venkman. Firebug's approach seems to be much simpler, ;-) but I left some detail unexplored since it was already very late.

The function that I quoted from firebug's source showed how sourceBox was created and added into the panel, but didn't show exactly how source lines are added into the panel. Nothing with <div> or <a> or <span> tags were mentioned in that function.

Just for the heck of it, I inspected the firebug again using the dom inspector, and found the clue of where the magic might have been. ;-)
this.SourceText = domplate(Firebug.Rep,
{
tag:
DIV(
FOR("line", "$object|lineIterator",
DIV({class: "sourceRow"},
SPAN({class: "sourceLine"}, "$line.lineNo"),
SPAN({class: "sourceRowText"}, "$line.text")
)
)
),

lineIterator: function(sourceText)
{
var maxLineNoChars = (sourceText.lines.length + "").length;
var list = [];

for (var i = 0; i < sourceText.lines.length; ++i)
{
// Make sure all line numbers are the same width (with a fixed-width font)
var lineNo = (i+1) + "";
while (lineNo.length < maxLineNoChars)
lineNo = " " + lineNo;

list.push({lineNo: lineNo, text: sourceText.lines[i]});
}

return list;
},

getHTML: function(sourceText)
{
return getSourceLines(sourceText.lines);
}
});
Notice the use of domplate function! It's a sweet little templating system ;-)

Tuesday, March 11, 2008

Javascript Source View II - Firebug vs Venkman

OK, last time, I was very happy to find out that venkman was using the XUL tree to represent the javascript source, while firebug was using html <a> tag and <span> tags.

As I was trying to find the script to load into the source view, I started to wonder how did those two software did it. I would imagine that we need to implement something that will
  1. load the file into a stream
  2. process the stream line by line

The findings from venkman actually amazed me once again, and along the way, I picked up a few other nice things ;-) So here it goes...

First, I was quite confident that I want to do it in xul tree way instead of the html way, so I started to scratch my head and buried my head into the venkman source code once again ;-) But before I went too far, I discovered that there are actually two different source views in venkman! One is called source, another one is called source2. source2 is the one that enables us to set breakpoints, and source doesn't do much, and it is hidden by default.
From venkman-views.xul
145
<!-- source2 view -->
146
147 <floatingview id="source2" title="&Source.label;" flex="1">
148 <vbox flex="1">
149 <tabbox id="source2-tabbox" flex="1">
150 <tabs id="source2-tabs">
151 <!--
152 We've got to put this placeholder tab in the tabs element to avoid
153 bogus strict warnings and exceptions.
154 -->
155 <tab id="source2-bloke" hidden="true"/>
156 </tabs>
157 <tabpanels id="source2-deck" flex="1"/>
158 </tabbox>
159 <textbox class="plain" id="source2-heading" readonly="true"/>
160 </vbox>
161 </floatingview>
162
163 <!-- source view -->
164
165 <floatingview id="source" title="&Source.label;" flex="1">
166 <vbox id="source-view-content" flex="1">
167 <toolbox>
168 <toolbar id="source-header" grippytooltiptext="&SourceHeader.tip;">
169 <label id="source-url" flex="1" crop="end"/>
170 </toolbar>
171 </toolbox>
172 <tree id="source-tree" flex="1" persist="width"
173 class="focusring"
174 onclick="console.views.source.onClick(event);"
175 onselect="console.views.source.onSelect(event);"
176 context="context:source">
177
178 <treecols>
179 <treecol id="source:col-0" width="20px"
180 display="&SourceCol0.display;" persist="hidden width"/>
181 <splitter class="tree-splitter"/>
182 <treecol id="source:col-1" width="50px"
183 display="&SourceCol1.display;" persist="hidden width"/>
184 <splitter class="tree-splitter"/>
185 <treecol id="source:col-2" flex="1" display=""
186 ignoreincolumnpicker="true" persist="hidden width"/>
187 </treecols>
188
189 <treechildren id="source-tree-body"/>
190
191 </tree>
192 </vbox>
193 </floatingview>
From venkman-views.js
2226 /*******************************************************************************
2227 * Source2 View
2228 *******************************************************************************/
2229
2230 console.views.source2 = new Object();
2231
2232 const VIEW_SOURCE2 = "source2";
2233 console.views.source2.viewId = VIEW_SOURCE2;
2234
2235 console.views.source2.init =
2236 function ss_init ()
2237 {

3527 /*******************************************************************************
3528 * Source View
3529 *******************************************************************************/
3530 console.views.source = new BasicOView();
3531
3532 const VIEW_SOURCE = "source";
3533 console.views.source.viewId = VIEW_SOURCE;
3534
3535 console.views.source.details = null;
3536 console.views.source.prettyPrint = false;
3537
3538 console.views.source.init =
3539 function sv_init()
3540 {

Notice that Source2 is an Object, while Source is a BasicOView. As what I usually do, I first scan the code for possible candidates of the key methods that I am looking for, and these two methods caught my attention.
3040 console.views.source2.loadSourceTextAtIndex =
3041 function s2v_loadsource (sourceText, index)
3042 {
3043 var sourceTab;
3044
3045 if (index in this.sourceTabList)
3046 {
3047 sourceTab = this.sourceTabList[index];
3048 sourceTab.content = null;
3049 sourceTab.sourceText = sourceText;
3050
3051 if ("currentContent" in this)
3052 {
3053 if (!ASSERT(sourceTab.tab,
3054 "existing sourcetab not fully initialized"))
3055 {
3056 return null;
3057 }
3058 sourceTab.tab.label.setAttribute("value", sourceText.shortName);
3059 sourceTab.iframe.setAttribute("targetSrc", sourceText.jsdURL);
3060 sourceTab.iframe.setAttribute("raiseOnSync", "true");
3061 this.syncOutputFrame(sourceTab.iframe);
3062 }
3063
3064 }
3065 else
3066 {
3067 sourceTab = {
3068 sourceText: sourceText,
3069 tab: null,
3070 iframe: null,
3071 content: null
3072 };
3073
3074 this.sourceTabList[index] = sourceTab;
3075
3076 if ("currentContent" in this)
3077 {
3078 this.createFrameFor(sourceTab, index, true);
3079 }
3080 }
3081
3082 return sourceTab;
3083 }
3085 console.views.source2.addSourceText =
3086 function s2v_addsource (sourceText)
3087 {

addSourceText calls loadSourceTextAtIndex to load a specific javascript source into source2's tabbed source views. addSourceText function will just happily take the sourceText, and stash it into a tab ;-) Great, the sourceText explains it all! (Grrr...) So what the heck is sourceText? For a while, I thought it was real string, but I quickly had an impression that I am wrong, because strings are not smart enough to be displayed into a xultree. So who calls the addSourceText? I dig on...
3456 console.views.source2.hooks["hook-display-sourcetext"] =
3457 console.views.source2.hooks["hook-display-sourcetext-soft"] =
3458 function s2v_hookDisplay (e)
3459 {
3460 var source2View = console.views.source2;
3461
3462 source2View.unmarkHighlight();
3463
3464 var sourceTab = source2View.addSourceText (e.sourceText);
3465 if (e.rangeStart)
3466 {
3467 source2View.highlightStart = e.rangeStart;
3468 source2View.highlightEnd = e.rangeEnd;
3469 source2View.highlightTab = sourceTab;
3470 source2View.markHighlight();
3471 }
3472
3473 source2View.scrollTabTo (sourceTab, e.targetLine, 0);
3474 }
So it is hooked by an event, which contains its own sourceText. OK, so what exactly does sourceText contain? I couldn't just do a find, since there are too many occurrence of sourceText in the code. Before I give it up for the day, I decided to have a close look at the source2 code, and guess what I found in the following function!
2466 console.views.source2.getContext =
2467 function s2v_getcontext (cx)
2468 {
2469 var source2View = console.views.source2;
2470
2471 cx.lineIsExecutable = false;
2472 var sourceText;
2473
2474 if (document.popupNode.localName == "tab")
2475 {
2476 cx.index = source2View.getIndexOfTab(document.popupNode);
2477 sourceText = source2View.sourceTabList[cx.index].sourceText;
2478 cx.url = cx.urlPattern = sourceText.url;
2479 }
2480 else
2481 {
2482 cx.index = source2View.tabs.selectedIndex;
2483 sourceText = source2View.sourceTabList[cx.index].sourceText;
2484 cx.url = cx.urlPattern = sourceText.url;
2485
2486 var target = document.popupNode;
2487 while (target)
2488 {
2489 if (target.localName == "line")
2490 break;
2491 target = target.parentNode;
2492 }
2493
2494 if (target)
2495 {
2496 cx.lineNumber = parseInt(target.childNodes[1].firstChild.data);
2497
2498 var row = cx.lineNumber - 1;
2499
2500 if (sourceText.lineMap && sourceText.lineMap[row] & LINE_BREAKABLE)
2501 {
2502 cx.lineIsExecutable = true;
2503 if ("scriptInstance" in sourceText)
2504 {
2505 var scriptInstance = sourceText.scriptInstance;
2506 var scriptWrapper =
2507 scriptInstance.getScriptWrapperAtLine(cx.lineNumber);
2508 if (scriptWrapper && scriptWrapper.jsdScript.isValid)
2509 {
2510 cx.scriptWrapper = scriptWrapper;
2511 cx.pc =
2512 scriptWrapper.jsdScript.lineToPc(cx.lineNumber,
2513 PCMAP_SOURCETEXT);
2514 }
2515 }
2516 else if ("scriptWrapper" in sourceText &&
2517 sourceText.scriptWrapper.jsdScript.isValid)
2518 {
2519 cx.scriptWrapper = sourceText.scriptWrapper;
2520 cx.pc =
2521 cx.scriptWrapper.jsdScript.lineToPc(cx.lineNumber,
2522 PCMAP_PRETTYPRINT);
2523 }
2524 }
2525
2526 if (sourceText.lineMap && sourceText.lineMap[row] & LINE_BREAK)
2527 {
2528 cx.hasBreak = true;
2529 if ("scriptInstance" in sourceText)
2530 {
2531 cx.breakWrapper =
2532 sourceText.scriptInstance.getBreakpoint(cx.lineNumber);
2533 }
2534 else if ("scriptWrapper" in sourceText && "pc" in cx)
2535 {
2536 cx.breakWrapper =
2537 sourceText.scriptWrapper.getBreakpoint(cx.pc);
2538 }
2539
2540 if ("breakWrapper" in cx && cx.breakWrapper.parentBP)
2541 cx.hasFBreak = true;
2542 }
2543 else if (sourceText.lineMap && sourceText.lineMap[row] & LINE_FBREAK)
2544 {
2545 cx.hasFBreak = true;
2546 cx.breakWrapper = getFutureBreakpoint(cx.url, cx.lineNumber);
2547 }
2548 }
2549 }
2550
2551 return cx;
2552 }
getContext is probably written for getting the right context menu. User got to right click on the script for it to show up, right? Hum... for a different line, say having a break point or not, it should probably show a different menu item. Therefore, it's gotta have something to do with the real sourceText! The code confirmed my theory! Line 2500 showed it all "sourceText.lineMap[row]". So it contains an attribute called lineMap, and lineMap contains the mapping of line number and actual code for this line!!! This is all cool, but still does explain how it is populated in the xultree... ;-( Oh, wait! Something else caught my attention... The checking of sourceText is to see if it conains a scriptInstance at line 2503, and or a scriptWrapper at line 2516
"scriptInstance" in sourceText

"scriptWrapper" in sourceText
leads me to believe that somehow, one of them or both of them are important when it comes to xultree.

Just a little side track, I found out that scriptWrapper is less than scriptInstance.
2506                     var scriptWrapper =
2507 scriptInstance.getScriptWrapperAtLine(cx.lineNumber);
Now I recall that I have read something about scriptWrapper and scriptInstance in the mozilla development wiki.
http://developer.mozilla.org/en/docs/Venkman_Internals#ScriptWrapper
And this led me to http://lxr.mozilla.org/mozilla/source/extensions/venkman/resources/content/venkman-debugger.js#914. The above document explains pretty clearly how scriptWrapper works. ;-) With the pointed out source, it's even more clear. So I will just continue to move on to my quest... How did the script get to be loaded inside a xultree.

I have a feeling that I dug a little too deep, and I must have missed something, since I am already down to the scriptWrapper level, which has already passed the point where the scripts are actually being loaded, so I went back to the declaration of the source2. This time, in detail... and I easy spotted out this array.
2248     this.cmdary =
2249 [
2250 ["close-source-tab", cmdCloseTab, CMD_CONSOLE],
2251 ["find-string", cmdFindString, CMD_CONSOLE],
2252 ["save-source-tab", cmdSaveTab, CMD_CONSOLE],
2253 ["reload-source-tab", cmdReloadTab, CMD_CONSOLE],
2254 ["source-coloring", cmdToggleColoring, 0],
2255 ["toggle-source-coloring", "source-coloring toggle", 0]
2256 ];
Notice the command that's called "reload-source-tab", and there is no command called "load-source-tab" ;-) Well, I don't quite care how come there isn't a "load-source-tab" method, as long as the "reload-source-tab" tells me how it is done, I will be happy ;-)

So I looked up cmdReloadTab, and in the very end of the function, I saw
sourceTab.sourceText.reloadSource(cb);
Well, we are back at sourceText again. ;-( Maybe we over looked at this eariler. Time to check out the sourceText. I tried and tried, I suddenly got use to using the mxr site, and found out how to find a function's declaration. Looking up code is becoming a breeze for me now. ;-)

Without much effort, I found the right SourceText's declaration in venkman-static.js. It is truly the starting point of the venkman ;-) (The all mighty console object is declared there ;-))
1108 function SourceText (scriptInstance)
1109 {
Of course, there is the
1191 SourceText.prototype.reloadSource =
1192 function st_reloadsrc (cb)
1193 {
Which calls
1313 SourceText.prototype.loadSource =
1314 function st_loadsrc (cb)
1315 {
1316 var sourceText = this;
1317
1318 function onComplete (data, url, status)
1319 {
1320 //dd ("loaded " + url + " with status " + status + "n" + data);
1321 sourceText.onSourceLoaded (data, url, status);
1322 };
1323
1324 if (this.isLoaded)
1325 {
1326 /* if we're loaded, callback right now, and return. */
1327 cb (Components.results.NS_OK);
1328 return;
1329 }
1330
1331 if ("isLoading" in this)
1332 {
1333 /* if we're in the process of loading, make a note of the callback, and
1334 * return. */
1335 if (typeof cb == "function")
1336 this.loadCallbacks.push (cb);
1337 return;
1338 }
1339 else
1340 this.loadCallbacks = new Array();
1341
1342 if (typeof cb == "function")
1343 this.loadCallbacks.push (cb);
1344 this.isLoading = true;
1345
1346 var ex;
1347 var src;
1348 var url = this.url;
1349 if (url.search (/^javascript:/i) == 0)
1350 {
1351 src = url;
1352 delete this.isLoading;
1353 }
1354 else
1355 {
1356 try
1357 {
1358 loadURLAsync (url, { onComplete: onComplete });
1359 return;
1360 }
1361 catch (ex)
1362 {
1363 display (getMsg(MSN_ERR_SOURCE_LOAD_FAILED, [url, ex]), MT_ERROR);
1364 onComplete (src, url, Components.results.NS_ERROR_FAILURE);
1365 return;
1366 }
1367 }
1368
1369 onComplete (src, url, Components.results.NS_OK);
1370 }
Great! see the critical function loadURLAsync? Oh, well, before I dig too deep and get so excited about not too closely related code again, I did also see the call back of onComplete. It turns out that onComplete did the dirty work of parsing the scripts by line and pack them into sourceText.lines.
1207 SourceText.prototype.onSourceLoaded =
1208 function st_oncomplete (data, url, status)
1209 {
...
1267 var ary = data.split(/rn|n|r/m);
1268
...

1287
1288 this.lines = ary;
...
Hum... How come no trace of xultree is found? This leads me to believe that my assumption is wrong. I went back and scan through all of the code that I have just looked, and arrived back at
3040 console.views.source2.loadSourceTextAtIndex =
3041 function s2v_loadsource (sourceText, index)
3042 {
3043 var sourceTab;
3044
3045 if (index in this.sourceTabList)
3046 {
3047 sourceTab = this.sourceTabList[index];
3048 sourceTab.content = null;
3049 sourceTab.sourceText = sourceText;
3050
3051 if ("currentContent" in this)
3052 {
3053 if (!ASSERT(sourceTab.tab,
3054 "existing sourcetab not fully initialized"))
3055 {
3056 return null;
3057 }
3058 sourceTab.tab.label.setAttribute("value", sourceText.shortName);
3059 sourceTab.iframe.setAttribute("targetSrc", sourceText.jsdURL);
3060 sourceTab.iframe.setAttribute("raiseOnSync", "true");
3061 this.syncOutputFrame(sourceTab.iframe);
3062 }
3063
3064 }
3065 else
3066 {
3067 sourceTab = {
3068 sourceText: sourceText,
3069 tab: null,
3070 iframe: null,
3071 content: null
3072 };
3073
3074 this.sourceTabList[index] = sourceTab;
3075
3076 if ("currentContent" in this)
3077 {
3078 this.createFrameFor(sourceTab, index, true);
3079 }
3080 }
3081
3082 return sourceTab;
3083 }
So this function is really trying to load source text into a tag with index defined by the index variable. If you take a closer look, you will find
3058             sourceTab.tab.label.setAttribute("value", sourceText.shortName);
3059 sourceTab.iframe.setAttribute("targetSrc", sourceText.jsdURL);
3060 sourceTab.iframe.setAttribute("raiseOnSync", "true");
So it is an iframe, and the file is loaded through sourceText.jsdURL. So it is not an xultree like other views, but an iframe!!! And of cource, there are utilities to create frame
console.views.source2.createFrameFor
and update margins
console.views.source2.updateLineMargin
etc, etc. Phew... My initial xultree assumption was wrong, and it is implemented using iframe.

Just to make the quest more complete, I am including how firebug is implementing the source text area ;-) This didn't take me nearly as long to figure out.

in debugger.js' ScriptPanel class, you will find a function called createSourceBox.
createSourceBox: function(sourceFile)
{
var lines = loadScriptLines(sourceFile, this.context);
if (!lines)
return null;

var maxLineNoChars = (lines.length + "").length;

var sourceBox = this.document.createElement("div");
sourceBox.repObject = sourceFile;
setClass(sourceBox, "sourceBox");
collapse(sourceBox, true);
this.panelNode.appendChild(sourceBox);

// For performance reason, append script lines in large chunks using the throttler,
// otherwise displaying a large script will freeze up the UI
var min = 0;
do
{
var max = min + scriptBlockSize;
if (max > lines.length)
max = lines.length;

var args = [lines, min, max-1, maxLineNoChars, sourceBox];
this.context.throttle(appendScriptLines, top, args);

min += scriptBlockSize;
} while (max < lines.length);

this.context.throttle(setLineBreakpoints, top, [sourceFile, sourceBox]);

if (sourceFile.text)
this.anonSourceBoxes.push(sourceBox);
else
this.sourceBoxes[sourceFile.href] = sourceBox;

return sourceBox;
},
OK! That was a long day for me... ;-)