[NetConsole]

Gregor.NetConsole

Sunday, April 27, 2003

A C#-like Interpreter

.NET Console is an interactive interpreter that can interpret a subset of the C# language. Using a parser and reflection, it evaluates statements, blocks, conditionals, loops, and simple functions.

.NET Console is implemented as a DLL, and an accompanying EXE. The DLL includes the parser/interpreter. An application passes strings to the interpreter for evaluation. There also is, however, an input loop, which can be used for transferring control to the interpreter, even if the app is not running on the console: Input and output are provided with an event mechanism, then.

The EXE is a thin console app wrapper arround the DLL. It starts .NET Console as an interactive console application. With the appropriate command line options, it can also be used for background processing.

Features

Here's what .NET Console can do:

Limitations

.NET Console can not:

Options

The code interpreter supports a number of options, which may be set either programmatically (by a host application) or interactively with special commands or library calls. Here is the list of options:

Note that options 'const' and 'explicit' are no longer supported. Every variable must be declared explicitly (but only once), either with the 'var' or 'const' keyword (const objects are actually read-only).

Scoping

Variables are valid only in the scope in which they are declared. There always is the global scope, which is retained between multiple evaluations. Variable hiding is illegal, except that local variables in user functions hide those in any caller's scope.

Outer scopes of nested user functions are determined according to the place of declaration, not the call stack.

Options may be scoped as well. For this, the routines of the UserOption module may be used. For example, it is possible to unset OptionConst for the scope of a user function.

Options set in a given scope hide options set in any outer scope. See above for how the outer scope is determined in nested user functions.

Imports may be scoped as well. Use the function of UserImport for this purpose.

As for scoping, the rules for options apply to imports as well.

Built-in Objects

There are several built-in objects, accessible by write-protected variables, whose classes are defined in the Gregor.NetConsole.User namespace, which allow interaction with the interpreter itself from code:

Note that the built-in objects are intended for use by interpreted code only, not for the any hosting application.

The Online Help

Type "show help", and you'll see this screen (subject to changes): 

.NET Console Help
=================
 
Information
 
  show help                     : get help
  show assemblies               : show loaded assemblies
  show namespaces               : show available namespaces
  show types [<filter-prefix>]  : show loaded types (optionally filtered)
  show variables                : show user-defined variables
  show members  <type> | <var>  : show members of a type
  show imports                  : show imported namespaces
  show options                  : show options
 
Basic Operations
 
  option <opt>                  : set options (see below)
  load <assembly>               : load an assembly into memory
  remove <var>                  : remove a variable
  clear variables               : remove all variables
  clear imports                 : remove all namespace imports
  exit                          : exit the console
 
C# Language
 
  using <namespace>             : import a namespace
 
  <expr>                        : evaluate an expression (see below)
  <expr> = <expr>               : assign an expression to another
                                  (left must be variable, field, or property)
 
Syntax Details
 
  opt  ::=    check[+|-]        : check imports against existing types 
            | const[+|-]        : assign to variables only once
            | explicit[+|-]     : variables must be declared with 'var'
            | nocase[+|-]       : ignore case of identifiers and commands
            | private[+|-]      : access private types and members
            | static[+|-]       : use static type information where possible
            | varargs[+|-]      : allow variable user function argument passing
            | verbose[+|-]      : show extended member information
 
  expr ::=    <literal>
            | new <type>()
            | new <type>[<length>]
            | <var>
            | <var>.<instance-member>+
            | <type>.<static-member>[.<instance-member>+]
 
  literal ::= <boolean-literal>
            | <character-literal>
            | <string-literal>
            | <verbatim-string-literal>
            | <integer-literal>
            | <hex-integer-literal>
            | <floating-point-literal>
            | <null-literal>
 
Remarks
 
  Nested arrays (ie., int[][]) cannot be created directly (use 
  System.Array.CreateInstance).
  For combining enum flags, calculate the value, and assign to
  value__ field.
  Overload resolution is dynamic: the actual object type is what
  matters; the closest match from all - including inherited -
  overload candidates is used.
  Floating point literals are parsed and displayed using invariant
  culture format (ig., 5.5), and are interpreted as System.Double.
  String literals support a few escape sequences (like \\, \n, \r,
  \t, \", \', \uNNNN, \xNN), as do character literals (which also
  support \') Verbatim strings (ig., @"str") are suppored as well.

Using .NET Console

.NET Console is capable of interpreting code that contains line breaks. Furthermore, when running it on the console, you can use a single trailing backslash for interactively editing input over multiple lines (also have a look at the CNetConsole.AllowMultiLineInput property).

Most examples simply list commands. If any output is listed, it's shown in dark blue color.

Example: Web Client

Run the EXE, and enter the following sequence of commands (press Enter after each line): 

load System.dll
using System.IO
using System.Net
sUrl = "http://peisker.net/"
req = WebRequest.Create(sUrl)
resp = req.GetResponse()
strm = resp.GetResponseStream()
rdr = new StreamReader(strm)
s = rdr.ReadToEnd()
rdr.Close()
s

Example: Using Meta Information

Here's how to get type information by reflection, and manage variables and namespace imports: 

obj = new object()
show members obj
show members System.Object
s = "Gregor"
delete s
s = "Gregor"
s
s = "Foo"
option const-
s = "Foo"
s
show variables
show assemblies
show imports
show options

Example: Expressions

System.DateTime.Now.ToString("dddd, MMMM d, yyyy", System.Globalization.CultureInfo.CreateSpecificCulture("en-US"))
s = new string['\\', 5]
int.Parse("ab", System.Globalization.NumberStyles.HexNumber)
System.Console.WriteLine("x is:\n 0x{0:x2}", 42)

Example: Arrays

using System
ai3 = new int[3, 5, 7]
System.Int32[3, 5, 7]
ai2 = Array.CreateInstance(ai3.GetType(), 1, 2)
System.Int32[1, 2][,,]
ai1 = Array.CreateInstance(ai2.GetType(), 11)
System.Int32[11][,][,,]
ai1[0] = ai2
System.Int32[1, 2][,,]
ai1[0][0, 0] = ai3
System.Int32[3, 5, 7]
ai1[0][0, 0]
System.Int32[3, 5, 7]
ai1[0][0, 0][0, 0, 0]
0

Here's how you can change the element type of an array: 

using Gregor.Core
ai = new int[3, 5]
System.Int32[3, 5]
ad = Conv.ChangeArrayType<int, double>(ai)
System.Double[3, 5]

Example: Enums

To combine enum flags, use code like this: 

flags = Bytes.CombineBitFlags<BindingFlags>(BindingFlags.Public, BindingFlags.Static);

Or make it more complicated: 

flags = BindingFlags.Default; // object must have the correct type
flags.value__ = Bytes.CombineBitFlags(BindingFlags.Public.value__, BindingFlags.Static.value__)

Example: Generic Types

using System.Collections.Generic
list = new List<string>()
list.Add("foo")
list.Add(1)

Example: Generic Methods

using Gregor.Core
using Gregor.Core.Collections
strings = new NCollection<string>
strings.Add("Foo")
aStr = Conv.CollectionToArray<string>(strings)

The call to the Sort() method in the following example chooses among both generic and non-generic methods, all being further overloaded by parameters, and having parameter arrays: 

using Gregor.Core
using Gregor.Core.Collections
tuples = new NCollection<ITuple<string>>()
tuples.Add(new CTuple<string>("Foo"))
tuples.Add(new CTuple<string>("Bar"))
sorted = Walk.Sort<ITuple<string>>(tuples, "Value1")

Note that .NET Console can not infer generic type arguments from the types of value arguments provided in the call.

Example: ref Parameters

dt1 = DateTime.Now
dt2 = DateTime.Now
Mat.Swap<DateTime>(dt1, dt2)

Example: Loops

foreach(c in "Foo"){System.Console.WriteLine(c);}
loop(i in new int[5]){System.Console.WriteLine(i);}

The loop construct simplifies typical counter loops, in case you need the counter variable (ig., when writing to array slots). It can be used with any ICollection (the counter goes from zero to ICollection.Count - 1) as well as integer (zero to the given value less one) expression - in the latter case, it works like a simple repeat command.

Example: Scoped Imports

foreach(var c in "Foo"){UserImport.Add("Gregor.Core"); Dev.Trace(c);}
Dev.Trace("Error.");

Example: User Functions

If no return statement is used, but the last statement in the block is used as the function's return value. 

foo(a, b){string.Concat(a, b, a);}
foo(1, 2)

A user function is treated like a variable ... 

foo

... which means you can pass it arround, like a function pointer or a delegate: 

using System.Collections
list = new ArrayList()
list.Add("foo"); list.Add("bar")
isFoo(s){object.Equals(s, "foo");}
filter(col, cb){ret = new ArrayList(); foreach(e in col){if(cb(e)){ret.Add(e);}} ret; }
filter(list, isFoo).Count

Another simple way for calling back code is passing an expression as a string parameter, and calling UserCommand.Evaluate - similiar to lambda expressions (although parameterizing the "lambda" would indeed require some messy string manipulation): 

using Gregor.Core
aStr = Ary.CreateArray("foo", "bar")
appendToElements(list, sExpr){loop(i in list){list[i] = string.Concat(list[i], ' ', UserCommand.Evaluate(sExpr));}}
appendToElements(aStr, "System.DateTime.Now")
Dev.TraceCollection(aStr)

The object-like nature of user functions allows for scoping and nesting as well (the last statements is an error): 

foo(){bar(){"Off the bar."}; bar();}
foo()
bar()

Example: Variable Arguments

With option varargs, you can enable passing an arbitrary number of arguments to a user function. In the user function, all arguments are available in the special "these" array (inspired by the "this" keyword), regardless of how the option is set. If variable arguments are enabled, named parameters are set to null to signal missing arguments: 

option varargs+
using Gregor.Core
Dev.TraceCallback = Dev.ConsoleTraceCallback
foo(a, b){Dev.Trace(a); Dev.Trace(b); Dev.Trace(these.Length); foreach(obj in these){Dev.Trace(obj);}}
foo(5)

Example: Event Handlers With User Functions

User functions can handle events of type System.EventHandler, or a type that differs from the latter only in that the second parameter type is derived from System.EventArgs. There are special helper routines for adding and removing user functions to and from events: 

using System
using Gregor.NetConsole
handleIt(sender, e){System.Console.WriteLine(string.Concat(sender, ": ", e));}
UserHandler.Add(AppDomain.CurrentDomain, "AssemblyLoad", handleIt);
load System.Data.dll
UserHandler.Remove(AppDomain.CurrentDomain, "AssemblyLoad", handleIt);

You can change the definition of an event handler user function anytime. .NET Console will always call the newest version (by name; note that overloading is not supported here). If you remove an event handler user function itself without de-registering it from the event, an error message will be logged (via Gregor.Core.Dev) evertime the event fires.

Expample: Callbacks Handled By User Functions

Here, a user function serves as a callback for the generic WalkFlags() routine of Gregor.Core.Bytes, which invokes a callback for each non-zero bit in an enum. Here's its signature: 

public static void WalkFlags<T>(T value, Callback<bool, T> cb);

T must be an enum, and a delegate returning bool and taking T is expected (note that the delegate type itself is defined as a generic type).

First, we define the user function - all it does is tracing the flag: 

using Gregor.Core
traceFlag(flag){Dev.Trace(flag); true;}
traceFlag(flag){...}

Next, we create a delegate that points to a compatible internal method which will ultimately invoke the user function: 

del = UserCallback.Create<Callback<bool, TypeRelationOptions>>(traceFlag)
Gregor.Core.Callback<System.Boolean, Gregor.Core.TypeRelationOptions>

Now, we can pass that delegate to Bytes.WalkFlags(Of T): 

Bytes.WalkFlags<TypeRelationOptions>(TypeRelationOptions.BaseTypes, del)
BaseClass
BaseInterfaces

Note that you can also use any compatible method that's not a user function as a callback - just create the delegate with a helper routine such as System.Delegate.CreateDelegate() or Gregor.Core.Reflect.CreateDelegate().

When you delete a user function, make sure you unsubscribe the delegate, in case you have registered it someplace. The delegates created by UserCallback.Create() are cached, and you can retrieve them with UserCallback.Get(), passing your user function or its name.

Example: Connecting Event Handlers

First, create a delegate: 

del = DelegateUtil.CreateInstanceDelegate<System.EventHandler>(target, "HandleIt")

Then, connect it to an event: 

EventUtil.AddEventHandlers(sender, "Click", del)

Example: User Objects

Just define the user functions you want to act as methods, and pass them to the official creation routine: 

foo(){"The Foo."}
uo = UserObject.Create(foo)
uo.foo()

For more complex objects, first write a user function as a factory, and use nested user functions for the methods: 

createPoint(x, y)
{
    ToString(){
        string.Concat('{', this.x, ',', this.y,'}');
    }
    ret = UserObject.Create(ToString);
    ret.AddProperty("x", x);
    ret.AddProperty("y", y);
    ret;
}

In this case, we have defined our own ToString method.

Note that the this pointer is available in those user functions that are assigned to the user object in the call to UserObject.Create only when invoked as a member of the user object. As in any user function, any actual argument is available in the these array, but that does never include this pointer. If you have defined your function outside of a factory function, you can use it in both contexts (as a user object member, or a free-standing function); you can find out in which context you're called using UserCommand.ExistsVariable("this").

You may also combine factory functions for some fuzzy object-based inheritance: 

createPoint3D(x, y, z)
{
    ToString(){
        string.Concat('{', this.x, ',', this.y, ',', this.z, '}');
    }
    ret = UserObject.Create(createPoint(x, y), ToString);
    ret.AddProperty("z", z);
    ret;
}

Here, two new user objects are created, and they're technically in a parent-child relationship, although logically you can treat the whole thing as just one object; the interpreter handles the details of method overriding and property lookup. Weird hacks based on that feature are possible, in that you can tag additional functionality to existing objects, and refer to several layers of one object, all depending one which reference you use.

The internal representation of user objects (class CUserObject) implements the ICloneable, IComparable, IDisposable, IEnumerable, IFormattable interfaces. If a user function with a name corresponding to a member of one of these interfaces is assigned to the user object, that method will be mapped to the interface method. In the following example, the user object may be passed to any code that expects a disposable object: 

Dispose(){System.IO.File.Delete("C:\\Con\\Con");}
uo = UserObject.Create(Dispose)

Don't try this at home.

Note that user objects do not support any other interface implementations (but see below for custom user object implementations by the host), or members other than methods and properties. On the upside, there is some late binding available (see InvokeMethod, GetProperty, etc.), and both methods and properties may be added dynamically (see AddProperty, AddMethod) - although you can not implicitly add a properties by assignment.

Late binding is in fact another way to achieve polymorphism - as long as several user objects have the same methods (by name), you can reuse such code as invokes these methods by name with CUserObject.InvokeMethod.

It is possible for an application to define its own user object implementations by deriving from Gregor.NetConsole.Engine.CUserObject. This way, additional interfaces may be supported, which are implemented by user functions. For example: 

public interface IUserInterface
{
    void Foo();
}

public class CUserObjectEx :
    Gregor.NetConsole.Engine.CUserObject,
    IUserInterface
{
    void IUserInterface.Foo(){
        if(this.ExistsMethod("Foo")){
            this.InvokeMethod("Foo");
        }else{
            throw new NotSupportedException("Interface not supported.");
        }
    }

}

For instantiating such an object, there are generic variants of UserObject.Create: 

UserObject.Create<CUserObjectEx>(Foo);

Running on the Console

You can run the .NET Console executable interactively, or use it for batch processing. For the latter, use the /eval:<expression> and/or /file:<file-path> options in any combination: they will be processed in sequence. You may also combine these options with the /run option, which fires up .NET Console in interactive mode after processing the expressions and files (if neither /eval: nor /file: are present, /run is implicit): 

NetConsole /file:Common.cs /eval:"option nocase+" /run

Note that quoting in a DOS box means "grouping", not "strings". You can use the backslash on most systems, however: 

NetConsole /eval:"s = \"Some string.\""

You can control where output is directed. Naturally, it goes to the console, but with the TraceCallback property of the Gregor.Core.Dev module, you can set a different target (by assigning a different handler) or even disable output (by assigning null).

Command line arguments can be accessed with UserCommand.GetCommandLine(), which returns a CCommandLine object (defined in Gregor.Core.Configuration). This way, code files can be parameterized.

See above for editing input over multiple lines with a trailing backslash.

You can interactively change the console colors used for input and output: 

using System
using Gregor.Core
Dev.OutputForeColor = ConsoleColor.DarkGreen
Dev.InputForeColor = ConsoleColor.Blue

Developing with .NET Console

Development Process Scenarios

Link an application against the DLL, to allow for in-depth diagnostics anytime. This is useful when an application has been deployed. The interpreter can even invoke private methods, and can show information about types and their members.

You can also run the console EXE, to find out how things work in .NET framework, and promptly see what the code does. It is often necessary to look beyound the documented behaviour. While not a substitute for test code, a readily available interpreter can actually encourage developers to get their hands dirty with the nitty-gritty details, instead of just making assumptions.

Another thing worth trying is intensive and interactive application and library testing. .NET Console complements debuggers with code variables management, enhanced type/member access, as well as a non-obstrusive windowing story. The latter is great for testing user interface code - provided .NET console is integrated into the application (for example, using the Gregor.AppCore.ShellControls.XConsole control).

Administrative Scenarios

You may also write special assemblies suitable for ad-hoc interpretation. This is a great option for administrative tasks that require flexibility: an object-oriented, interactive shell for command line enthusiasts.

Scripting-like batch processing is an obtion as well: use C#-like syntax for everyday tasks without compilation.

Hosting Scenarios

Finally, .NET Console is easy to integrate into any application that requires expression evaluation (for example, in templates or formulas), user scripting or simply a greater level of user customization. For example:

Implementation Details

Layers of a Code Interpreter

.NET Console performs its task in three steps: tokenizing, syntax analysis, and evaluation.

The tokenizing is provided by Gregor.Core.CCurlyTokenizer, which is able break any C-like language source into a list of tokens. Have a look at the Gregor.Core.TokenType enumeration to get an idea of the level of analysis performed.

For syntactical analysis, the classes in Gregor.NetConsole.Parser are used. There is a model for a syntax tree that includes statements, control structures, method definitions, and a number of expressions. Note that this is not a complete model of C# syntax.

The evaluation is performed against the syntax tree. The implementation is shared between the classes in Gregor.NetConsole.Engine, most notably, CCurlyInterpreter, and the latter's base class, CInterpreter. Whereas CCurlyInterpreter works closely with the syntax model and is specialized on C#, CInterpreter provides more general evaluation services, such as reflection helpers, assembly and namespace support, and scope and variable management. CInterpreter is potentially usable as a base class for other, similiar interpreters running against the .NET framework.

The interpreter is wrapped by the CNetConsole class, which contains the input loop mentioned above, and pre-evaluates certain commands that are mostly not in the C# specs, but useful for an interactive interpreter (loading assemblies, showing help, getting type information).

The Tokenizer

[to be supplied]

The Syntax Model

Here are some hints at the code model entities produced by the parser:

Note that the parser cannot completely create the syntax tree, since it is supposed to be ignorant of the type environment. Therefore, some (not all) type reference nodes are created not by the parser, but by the engine (interpreter), taking any imports into account. In particular, this happens before a chained expression is evaluated.

The close relationship between statements and expressions that's typical for C-based languages is modelled not by inheritance, but by containment: A statement contains at least one expression (for now, that would be a call chain), and an expression must have a statement as a direct of indirect parent somewhere up the syntax tree.

A further simplification is that variables may be introduced anywhere an expression is used: a variable declaration is in fact an expression (see the class hierarchy below): 

var ai = new int[var len = 5];
loop(i in len){ai[i] = i;}

For a better understanding of the syntax model, run the TestNetConsole2 project of the Gregor.NetConsole solution; it has some tracing code that prints out the syntax tree. Or, run the following code statements in .NET Console: 

using Gregor.Core
using Gregor.NetConsole.Parser
Dev.TraceCallback = Dev.ConsoleTraceCallback
tokens = new CCurlyTokenizer().Tokenize("5.GetType()")
tree = new CCurlyParser().Parse(tokens)
Dev.TraceTree(tree)

The output: 

Gregor.Core.Collections.CTreeWalker
  CCodeRoot {TokenIndex:0,TokenCount:5,ChildNodeCount:1}
    CStatement {TokenIndex:0,TokenCount:5,ChildNodeCount:1}
      CChainedExpression {TokenIndex:0,TokenCount:5,ChildNodeCount:2}
        CLiteralExpression {TokenIndex:0,TokenCount:1,ChildNodeCount:0}
        CMethodCallExpression {TokenIndex:2,TokenCount:3,ChildNodeCount:1}
          CMemberReference {TokenIndex:2,TokenCount:1,ChildNodeCount:0}

Here's another example of a parse tree: 

// WebEditApp.MainForm.ConsoleForm.Console.Log(new string('#', 5))
CCodeRoot
    CStatement
        CChainedExpression
            CTypeReference
            CVariableExpression
            CVariableExpression
            CVariableExpression
            CMethodCallExpression
                CMemberReference
                CStatement
                    CChainedExpression
                        CNewExpression
                            CTypeReference
                            CLiteralExpression
                            CLiteralExpression

For contrast, here's the parser class hierarchy: 

System.Object, Gregor.Core.Collections.ITreeNode
    CCodeNode
        CBlock
        CBlockHeader
            CControlHeader
            CMemberHeader
        CCodeRoot
        CExpression
            CChainedExpression
            CSimpleExpression
                CCallExpression
                    CIndexerCallExpression
                    CMethodCallExpression
                    CNewExpression
                        CObjectNewExpression
                        CArrayNewExpression
                CLiteralExpression
                CVariableExpressionBase
                    CVariableExpression
                    CVariableDeclaration
        CMemberReference
        CStatement
        CTypeReference

Every non-leaf class is abstract. This makes using a type code property feasible (NodeType property), which in turn is helpful in violating fundamentalist OO principles (No switches! All logic done by virtual dispatch!), which in another turn helps with the worthy goal of separating the syntax model classes from the interpreter.

The Interpreter

To avoid endless loops, there is a limit currently defaulting to 10,000 iterations per loop (an exception is thrown on reaching the limit). It can be changed with the CInterpreter.LoopLimit property.

Likewise, the StackLimit property (currently defaulting to 1,000 block nestings) guards against unbounded recursion and stack overflow.

Internally, user functions are represented by instances of class Gregor.NetConsole.Engine.CUserFunction.

[more to be supplied]