Category Archives: GarageGames

VSTorqueScript #6

I've spent the last two days working on the Debug Engine and I've made some progress:

  • Tracepoint Support
  • Breakpoint Support
  • Support for the Watch window (I might have enlisted this already once but now support is more complete)

I'm not going to post any screenshots this time, instead I'll try to create an alpha package for you to test. This will probably take some time, too, because I want to make sure that you can install and uninstall it cleanly. I've been using the experimental hive so far, so I have to make sure it doesn't corrupt anyone's installation accidentially... :-)

A few implementation notes for interested people as always:

  • If you ever have to marshal an object to an IntPtr, be very careful, because doing the wrong thing will certainly crash your application - I had that happen to me and it took me quite some time to figure out why Visual Studio was suddenly  'experiencing internal problems' and had to be closed' all the time:
    When you have to convert an IntPtr to a wrapped object, use

    MyObject = (MyType) GetObjectFromIUnknown( intPtr ); // or As MyType;
    

    As you see, the intPtr is a COM object and every COM object implements IUnknown, so the call is pretty safe. The conversion afterwards is already 'managed', so safe, too.
    If you want to convert an IntPtr from an object, you have to be more careful though. The Marshal namespace offers at least two methods for it:

    • public static IntPtr GetComInterfaceForObject(
      	Object o,
      	Type T
      )
      
    • public static IntPtr GetIUnknownForObject(
      	Object o
      )
      

    Don't use the latter for anything except if you really need an IUnknown interface IntPtr pointer!
    The method really only returns a pointer to the IUnknown interface of the object, which usually isn't what the unmanaged code expects - which probably will result in misinterpreting some uninitialized memory as a vtable that is not there.
    You have to use GetComInterfaceForObject and don't forget this. Because you're only working with IntPtrs in the end, there is nothing to warn you about your misstep.

  • If you want to use an object with System.Collection.Generic.Dictionary<K,V>, it's not sufficient to implement IEquatable<K> as stated in the documentation - you also have to override GetHashCode (and override the normal Equals, too, then while you're at it), because Dictionary uses GetHashCode for the bucket assignments and you can implement IEquatable as hard as you want, GetHashCode might simply assign two 'equal' objects to different buffers and you have a problem..
  • One of the benefits of COM is that no one cares about your implementation.. take advantage of it!
    I had to remember this the hard way, when I implemented my breakpoint code. At first I wrote a PendingBreakpoint class, which implemented IDebugPendingBreakpoint2, and then a BoundBreakpoint one, which implemented IDebugBoundBreakpoint2. It was unnecessary messy and today I sighed and rewrote the breakpoint code to use just one Breakpoint class that implements both interfaces and it's a lot more elegant now.
  • Funny fact: An enum that is even mentioned in MSDN and the SDK docs is missing from the Interops library:
    public enum enum_BP_UNBOUND_REASON
    {
        BPUR_UNKNOWN = 0x0000,
        BPUR_CODE_UNLOADED = 0x0002,
        BPUR_BREAKPOINT_REBIND = 0x0003,
        BPUR_BREAKPOINT_ERROR = 0x0004
    };
    
  • The initialization process for the DE is as follows:
    1. DebugEngine and Program Creation Events
    2. Load Complete Event
    3. Call CreatePendingBreakpoint for all known breakpoints
    4. First Continue Call
    5. Entry Point Event
    6. Either the second Continue call or the SDM enters break mode

I had to write a few wrapper classes, too, which I might post about next time around. It's enough for today though.

Cheers,
Black

VSTorqueScript #5.1

Sadly there isn't a lot new that I can post since I've hardly had time to work on my pet project during the last week - a few friends from Texas and California came over for the Oktoberfest and it suffices to say that I drank a lot of beer last week.

Nonetheless there is some progress and I've finished support for stepping in break mode (although I still have no idea what enum_STEPKIND.STEP_BACKWARDS is about, since I've never heard of stepping backwards before).

There is no screenshot this time because you wouldn't really see a difference, except if I did, before/after screenshots to show that the stepping works.

A few coding notes as always:

  • The current TorqueScript telnet debugger doesn't really tell you whether it breaks because of stepping or because it has hit a breakpoint, so I have to figure it myself. It is pretty straightforward though because stepping usually implies a certain callstack change (or the lack thereof) when finished:
    • "Step Into" implies that the new callstack height > old height
    • "Step Over" implies no change
    • "Step Out" implies of course: new callstack height < old height

    So it's pretty easy to figure out whether the stepping was finished or interrupted by a breakpoint.

  • The class diagram designer/viewer in Visual Studio 200x is awesome. If you haven't tried it, you've missed a really helpful feature of Visual Studio.
    Lately I've been using it to comment my public functions and parameters, etc. and for small refactorings (like name changes) in general.

Last but not least I think I've written a neat communication class for synchronous and asynchronous message processing which I'd like to share (if you're going to use it though, please tell me and/or send me some feedback):

Read more »

VSTorqueScript #5

It took me quite long to make progess in the last days. Partially this is due me being in Munich again and working in my uni job part-time again - and I'm stunned each time how tired I am when I come home.

It also is due to me experimenting with my TelnetDebuggerClient implementation that connects my debug engine to Torque's Telnet Debugger. I've tried quite a few different approaches for synchronous and asynchronous communication before I could settle for the simplest and the more advanced ones reside in the repository's history. I'll probably have to switch to them sooner or later because I only implement asynchronous communication at the moment which is fine for most things except when you need more advanced debugging info like for displaying objects properly in the debug window for example.

I've also had to read the "Definitive ANTLR Reference" in one night because I had to return it to uni's library the next day, so I was busy for 7 hours reading 350 pages of a parser reference. It was interesting though and tonight I'll try to write a QuakeC grammar - that's the plan at least.

However yesterday, I had time to work on the debug engine some more and here's a picture that tells it all:

VSTorqueScript now supports displaying the callstack, too

VSTorqueScript now supports displaying the callstack, too

As you see you can now break a program and it will retrieve the callstack from Torque and update the callstack window automatically and open the right file and jump to the correct file position, too - or rather as I've just noticed one-line off because VS line indicies have base 0 and TorqueScript uses base 1 >_>...

A few implementation notes as always:

  • I t seems, if your Debug Engine doesn't implement IDebugEngineLaunch2, your Program Provider has to implement IDebugProgramProvider2.WatchForProviderEvents
  • Moreover, if you don't implement IDebugEngineLaunch2, which in turn forces you to implement IDebugProcess2 (and maybe IDebugProcess3, too), suddenly some "deprecated" methods in your Program class (your IDebugProgram2 implementation) will be called, which, if you follow MSDN and the debug engine sampe, will result in failed asserts and the nice fail message "This function is not called by the debugger." :-)
    It's kind of logical that IDebugProgram2.Execute() is going to be called because the debugger has no IDebugProcess2 interface to call Execute on..
  • What is enum_FRAMEINFO_FLAGS_VALUES? It's undocumented and I suspect it's used for the undocumented m_dwFlags field in FRAMEINFO...
  • I'm totally stunned by how stable Visual Studio is. Most the interfaces I implement are only half implemented and return E_NOTIMPL on most calls and Visual Studio runs and runs well and only displays the information it can in a clean way.
    I guess this is one of the advantages of the COM programming/design paradigma and I think it's great. Being able to implement interfaces bit by bit and add functionality slowly makes debugging easier and also helps you understand things better - which is good because even though a lot is documented, you still need some practice and trial & error to get things right eventually.

Enough is enough though for now, and I'll head to uni in a bit.

Only one last remark: I've started reading the book "The Pragmatic Programmer" and so far it's really entertaining and, I guess, good.
I'll write more on it, when I've finished it and can actually assess its qualities..

Cheers,
Andreas

VSTorqueScript #4

My little pet project is actually called VSTorqueScript, so I've decided to use the correct title for my posts, too.

Today has been a somewhat productive day, although not as productive as I've hoped, because I was stuck with a very stupid issue for 3 or 4 hours. I'm not proud of it, but it made me go to the VSX forum and become active there.

I've worked more on the debug engine afterwards (well, actually I went jogging and swimming - in that order.. think about it ) and made some progress. Quite a bit of the stub stuff is implemented now and I think I'll start implementing the real thing tomorrow.

No screenshots this time, I'm sorry, but for the SDK users, I have a little helper file. If you look at the managed debug example, you'll find an AD7Events file that wraps the debug event interfaces.
I wasn't easily able to port it over, so I decided to write my own version of it (while keeping the comments).

I've appended it to the post, if you want to take a look.

On other news I'll probably switch this blog over to my own webspace soon to get some advantages (like expandable sourcecode amongst things), but I think I'll stick to wordpress.com for a few more weeks.
It's really a pity that you can't redirect from wordpress to your domain, but only vice-versa :(

Anyway, off to work some more on my code.

Cheers,
Andreas

Read more »

Visual TorqueScript #3.75

Another minor update, which I'm mostly using to publish a useful patch.
I've forgotten to mention that I've already created a custom settings page for my Visual TorqueScript projects. Yesterday night (before going to bed) I decided to add a configuration dependent page, too.
As you'll see it's very Torsion-esque, which is intentional and I plan to support .torsion files directly (as project files) later-on.

The main reason is that Torsion rocks and there is no reason to have yet another project file format for TorqueScript projects.

I hope that it will also be an incentive for people to give my plugin a try (if I ever release it).

A few notes (programatically so to speak :D ):

  • Again the managed project framework is awesome and once you figure out where to look for things you'll certainly find eternal bliss, too..
  • I generally derive from SettingsPage directly, because I don't want any of the regular faff in my project.
  • Don't forget to publish your pages via ProvideObject (or ProvideClass from my previous post)
  • You don't need to create your own GUIDs for everything. Classses are automatically provided with a unique GUID - try it and access typeof(SomeClassWithoutGuidAttribute).GUID and it'll still work.
    (If I said something wrong, please correct me..)
  • SettingsPage has an IsDirty property which you have to set, when a property is changed (to update the Apply button state).
    It results in pretty ugly copy'n'paste code, because you can't use default property accessors anymore.
    To fix this I spent an hour or two and I've come up with an elegant (in my opinion) patch to fix the issue:

    Index: ProjectBase/SettingsPage.cs
    ===================================================================
    --- ProjectBase/SettingsPage.cs	(revision 10)
    +++ ProjectBase/SettingsPage.cs	(working copy)
    @@ -27,7 +27,26 @@
    
     namespace Microsoft.VisualStudio.Package
     {
    +    // Added to make it easier to edit pages [9/10/2008 Andreas]
    
    +    class SettingsPagePropertyDescriptor : DesignPropertyDescriptor
    +    {
    +        public SettingsPagePropertyDescriptor(PropertyDescriptor prop)
    +            : base(prop)
    +        {
    +        }
    +
    +        public override void SetValue(object component, object value)
    +        {
    +            if (component is SettingsPage)
    +            {
    +                SettingsPage page = (SettingsPage)component;
    +                page.IsDirty = true;
    +            }
    +            base.SetValue(component, value);
    +        }
    +    }
    +
         /// <summary>
         /// The base class for property pages.
         /// </summary>
    @@ -78,7 +97,7 @@
                 get { return this.grid; }
             }
    
    -        protected bool IsDirty
    +        protected internal bool IsDirty
             {
                 get
                 {
    @@ -203,6 +222,13 @@
    
             #endregion
    
    +        #region overriden methods.
    +        public override DesignPropertyDescriptor CreateDesignPropertyDescriptor(PropertyDescriptor propertyDescriptor)
    +        {
    +            return new SettingsPagePropertyDescriptor(propertyDescriptor);
    +        }
    +        #endregion
    +
             #region IPropertyPage methods.
             public virtual void Activate(IntPtr parent, RECT[] pRect, int bModal)
             {
    @@ -410,6 +436,8 @@
                     try
                     {
                         Marshal.WriteIntPtr(ppUnk, p);
    +                    // Reset IsDirty before rebinding all properties [9/10/2008 Andreas]
    +                    IsDirty = false;
                         this.BindProperties();
                         // BUGBUG -- this is really bad casting a pointer to "int"...
                         this.grid.SetSelectedObjects(1, ppUnk.ToInt32());
    

    It also contains a one-line fix to reset IsDirty when reloading the page or switching configurations (ie. any time the properties are reset to their old value).

Cheers,
Black

Visual TorqueScript #3.5

Small update: After hours and hours of error and trial (and implementing a few interfaces by using Copy&Paste wisely), I've finally managed to get Visual Studio to load my debug engine.

This looks gorgeous, doesn't it?

A few notes for those interested in writing Debug Engines:

  • See my previous post for the ProvideDebugEngine helper attribute
  • There exists a really nice managed debug engine example, that isn't part of the SDK.
    You can find it here.
  • You are required to set a PortSupplier, even if the documentation says otherwise (this was a major source of frustration in the last hour).
  • Since ProvideDebugEngine requires a Type for PortSupplier, I've been using this stub class:
        /// <summary>
        /// The class is just a stub for use with ProvideDebugEngine.
        /// It seems it's mandatory to provide a PortSupplier GUID with a debug engine.
        /// </summary>
        [Guid("708C1ECA-FF48-11D2-904F-00C04FA302A1")]
        internal sealed class DefaultPortSupplier
        {
        }
    
  • I've also added another attribute (ProvideClass) for convenience to register a class directly (it's the same as using ProvideObject on the current class type).
    Since the code is quite short, I'm going to supply it directly:

    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    
    namespace Microsoft.VisualStudio.Shell
    {
        /// <summary>
        /// This is just a wrapper for ProvideObject to provide the class the attribute is used on
        /// </summary>
        [AttributeUsage(AttributeTargets.Class, AllowMultiple = false, Inherited = true)]
        class ProvideClass : RegistrationAttribute
        {
            public override void Register(RegistrationAttribute.RegistrationContext context)
            {
                ProvideObjectAttribute objectProvider = new ProvideObjectAttribute(context.ComponentType);
                objectProvider.Register(context);
            }
    
            public override void Unregister(RegistrationAttribute.RegistrationContext context)
            {
                ProvideObjectAttribute objectProvider = new ProvideObjectAttribute(context.ComponentType);
                objectProvider.Unregister(context);
            }
        }
    }
    

And yes, I love the tag :-)
Cheers,
Andreas

Visual TorqueScript #3

I've spent a part of the last two days cursing the managed project framework (it's a bliss, too, and I'm thankful that it's available at the same time), because it is too different from what I need, and I'll probably end up rewriting most of it to use different approaches and not use MSBuild anymore.

I've spent more time, however, on reading up on Debug Engines in Visual Studio 2008 and I've started work on my own yesterday. To be honest, I haven't really started to work on it, I've rather spent the evening and most of today figuring out how to register a debug engine at all and write attributes for that, so that it becomes an automated process, just like it's done for Packages and Language Services by using attributes like ProvidePackage and ProvideLanguageService.

The code has turned out pretty nicely and now you can use the attributes like this, for example, to register a debg engine:

    [ProvideDebugEngine("{5887F58F-3B7F-401d-95CE-A7BADA1E4D7D}", "TorqueScript Debug Engine",
        Attach = true, ProgramProvider = typeof(VSTorqueScript_DebugEngine))]
    [ProvideIncompatibleEngineInfo( "{5887F58F-3B7F-401d-95CE-A7BADA1E4D7A}" )]
    [ProvideIncompatibleEngineInfo( "{5887F58F-3B7F-401d-95CE-A7BADA1E4D7F}" )]
    [ProvideAutoSelectIncompatibleEngineInfo( "{5887F58F-3B7F-401d-95CE-A7BADA1E4D7F}" )]
    [ProvideObject(VSTorqueScript_DebugEngine)
    [Guid(GuidList.guidVSTorqueScript_DebugEngineString)]
    public sealed class VSTorqueScript_DebugEngine
...

In my opinion this looks pretty neat - and it works.. \o/
I really love the whole concept of attributes and the way they make coding a lot easier (even though it's hard to find information about useful attributes, as you only find out they exist when you stumble over them or write them on your own).

You can find the code here: http://www.icculus.org/~black/weblog/VS_DebugEngine/

A few notes:

  • The implementation was done making heavy use of copy&paste and quick replace (I copied the table containing the help description of the metrics into the source file and reworked it using RE magic to look the way it does now)
  • At first I used Type.GetProperties to enumerate all properties and write them out into the registry but that bloated the resulting registry entry, of course. Because of this I later switched to the HashTable approach that I first found in SDK's implementation of ProvideLanguageServiceAttribute.
  • The restriction that all attribute parameters must be constant expressions forced me to make a few changes - at first I had Guid properties and for the incompatible engines list I expected a Guid[], which didn't work. I changed the Guids to Strings and added the Provide*EngineInfo attributes instead of the incompatible engines list (same for the autoselect one, of course).
  • The Provide*EngineInfo attributes contained their registry code at first. This was ugly because they kind of required a ProvideDebugEngine attribute to exist, too, which I couldn't enforce. Later I changed them to be pure value holders and moved all registry code into ProvideDebugEngine.
    Now nothing happens if you don't supply a ProvideDebugEngine attribute, too.

Now I'll try to start to work on the actual Debug Engine for TorqueScript.

Cheers,
Andreas

Visual TorqueScript #2

I haven't found time to add IntelliSense to the language service yet, so I've only added a basic parser and enabled brace highlighting/matching which was easier to do.

I've been told that a working debugger would be the most important feature, so I try to get started with it, but before I can do that I have to add a custom project type to support the standard .cs extension for TorqueScript and to offer a specialized debug options property page for Torque's telnet debugger.

Here you can see the progress I've made so far:
I've added a new custom project type and changed everything to support TorqueScript files that use the usual .cs extension. There is no conflict with C# whatsoever. (At least from what I know..)

Next I'll look into removing the MSBuild faff (if possible at all) and adding the debug configuration page.

Last but not least I've found another neat tool that I can only recommend and want to share with whoever might read this:
It's called VSSDK Assist and it's a so-called Guidance Package - it provides you with a neat assistance window that helps you do common tasks using the Visual Studio SDK in the preferred way (it supplies you with lots of dialog-based wizards for all kinds of things like custom pages/tools/editors/projects/...).

You can grab it here.

More to come.
Cheers,
Black

Visual TorqueScript #1

I've spent most of this week learning as much about Visual Studio's extensibility features as possible and I've decided to start a tiny project to see how easy it is to add a new language service to Visual Studio.

My conclusion so far is: it's not easy and that obvious but it certainly is easier than writing your own IDE. There is also the advantage that you have access to a nice debugging environment if you manage to write your own debugger service.

Anyway, I've started to work on adding basic support for TorqueScript to Visual Studio and after much learning, I've finished implementing the colorizer for TorqueScript today:

As you see I have to use the extension .tscript for now, because as you know .cs is already reserved for C#. I'm going to fix this later by adding a TorqueScript project type that overrides this link locally for your TorqueScript projects (I could live with that).

Creating your own language services is pretty easy, if you are ok with using C# and ManagedBabel. You just have to supply the specific grammar files for the managed lexer and parser generators and you're pretty much done with the basic stuff.

The Visual Studio SDK is pretty neat (you can grab it from here if you want) and another neat thing no one has probably heard about is that Microsoft gives away the Visual Studio Shell for free - yeah for free. That means you can build your own custom tools that have the Visual Studio look and feel and you don't have to pay Microsoft a dime for it.

A good place to start in general if you're interested is the VS Extensibility Center.

Stay tuned for more features (syntax parsing and debugging featues are the next thing on my plan).
Cheers,
BlackHC

XML, XSLT, HTML and what it has to do with Torque GameBuilder

Good title, eh?
But really, it seems like a lot of interesting buzz words were thrown together to create a leet-looking tagline and yes, that is my original intention.

On the other hand I really had to do with all those buzz words, simply because I was too lazy to start up Doxygen...
You see, Torque GameBuilder(or short TGB) didn't use to have very good documentation or rather the function and class reference was always at least a little bit obsolete, so when I needed a good and fast-loading reference, I preferred my self-created .chm file over GarageGames' pdf reference - which was also awkward to navigate in. TGB has console functions that output all console functions and classes in doxygen-friendly C++-parser compatible data and you can feed that to doxygen and create a nice reference from it, so that is pretty simple job and gives you a very useful custom reference nevertheless.

But as it comes I'm now using Torsion(a nice little IDE for TorqueScript) and it has its own way of extracting the symbol information from TGB and storing it in an .exports file, which is nothing else but a renamed .xml file. XML stands for Extended Mark-up Language and is the generic big brother of HTML, if you haven't heard of it.

Knowing this and being in dire need of a new reference file, I thought "Heck, I'm too lazy to take the 10 minutes and 5 relatively-easy steps to create my usual doxygen documentation for the new release - no, I'm somehow going to use the XML data to create my documentation! And although I don't know anything about XML in general or any tools for working with it, it certainly will be a lot more entertaining, time-saving and fun than the doxygen way!".

Well, sometimes I'm as dumb as you can get - but not this time! Yes, it drained all my resources for half a day and yes,  eventually I started up doxygen and did it the usual way, but... - ok, lets rephrase that: Although I haven't been able to use it productively after all, I've learnt a lot and it certainly will help me some other time in my life. *puh*

As I'm owning Visual Studio .NET, I was convinced that it would offer some way to edit XML data and represent it in a human-readable way.
I've used Torsion's exports xml from a free TGB game (you can download it from www.garagegames.com) called Marble Tactices and here is a little excerpt:

<class>
<name>ActionMap</name>
<base>SimObject</base>
<method>
<name>bind</name>
<args> device, action, [modifier spec, mod...], command </args>
</method>

This isn't exactly readable and after opening it in VS.NET I spend a long time trying to figure out how to control its representation. After approximately half an hour I stumbled over the XML stylesheet language or more correctly: XML stylesheet language transformations (XSLT).

Here's the same excerpt from above after it has been transformed by one of my XSLT files (or rather by msxsl with one of my XSLT files specified as stylesheet):

ActionMap   SimObject

  • Methods
    • bind( device, action, [modifier spec, mod...], command )

XSLT is an XML-based format that can be used to describe the conversion from one XML dataset to another. HTML or rather so-called well-formed HTML(HTML that adheres to the XML standard) can thus be seen as the output dataset and consequently you can use XSLT to describe the conversion process that transforms the content of a certain type of XML files into concise and human-readable data (human-readable as in "pleasant to the reader's eye").

Microsoft's help is very neat and I only took me half an hour to have the first basic version working and it probably took longer to learn HTML and CSS (knowledge which I haven't actually used for my final PoC) than to understand XSLT as concept thanks to very good examples and good documentation. Talking about documentation, if you need to learn HTML, XML, CSS or any other web-related standard, W3C is the best place to find specifications and help.

Well, maybe it's just me but it's a lot easier to learn something new comprehensively if you read the specs rather than some whacky tutorials for beginners that you can find on the net everywhere...

Anyway, XSLT is great, great and again great like a lot of things that have to do with XML since they are very easy to use. XML really seems like a good way to go, since there are many nice generic tools available for it and one doesn't have to reinvent the wheel all the time.

If you find time to look at my XSLT "code" and the output it's capable to create, you'll realize what a time-safer it could become, if XML becomes more widely used (in games programming).

Writing all 3 XSLT PoCs/tests took me a bit over half a day and finding, learning and getting mktree to work properly took one third of that time, so you see that XSLT really has got a nice learning-curve. For those who don't know mktree, it is a free javascript library that you can use the create tree structures in your HTML/DHTML files.

You can find and download my Torsion exports -> HTML XSLT example (3 proofs of concept) here.
BTW Everything else of importance is explained in the readme.txt.

Enjoy and stay tuned,
 Andreas Kirsch