Writing a driver for a non ADO data source.

Hugo · April 2012

Hi;

Im trying to write a driver for a non ADO data source - by which I mean I have data, in memory already loaded and the data is exposed simply as IList[T] objects.

In our data store we expose objects of type Cursor[T] this class implements a property "Objects" that implements the IList[T] interface and so obviously can be queried easily.

I can automatically and dynamically create instances of Cursor[T] objects if I need to, but I cant see how to get this to work with the LINQPad driver model.

It seems that the driver model insists on there being a DataServiceContext object that has members of type DataServiceQuery[T] - it also looks as if LINQPad creates and compiles this class itself from metadata based on the schema etc of the DB.

No doubt under-the-hood it scrutinizes this DataServiceContext object at runtime to get at the various DataServiceQuery[T] members and so is then able to query against their sequences.

It's all a bit puzzling and I am wondering how I can get this to work.

DataServiceQuery[T] instances can only be created via the CreateQuery[T] method in DataServiceContext but this requires some form of string URI and this too is confusing me.

I know nothing about DataServiceContext or DataServiceQuery stuff, never used it - and never needed to with LINQ fo Objects either so this is pretty puzzling.

Must we expose DataServiceQuery[T] members or can we just expose something that is IEnumerable[T] and the driver will "know this"?

Thanks for any guidance.

PS: I cant use angle brackets for generics in this editor!

Hugo · April 2012

OK I did some research on this - yes I can simply create properties in the DataServiceContext that return IEnumerable[T] - LINQPad runs fine with this.

But there are two new problems now:

1. The assembly that defines my types (for the data) must be signed - it seems so anyway.
2. The build/compile process on the driver seems to need this assembly to be a disk file.

At leatst I'm progressing...

Hugo

Hugo · April 2012

In addition - the query results rendering includes all public properties and fields - is there a way to restrict this? any attributes or any options in LINQPad?

JoeAlbahari · April 2012

Only the assembly that contains your driver needs to be signed. Obviously, if this statically references other assemblies, those will need to be signed, too.

Regarding filtering the output, there are a number of ways to do this. It's described fully in the documentation here:
http://www.linqpad.net/extensibility.aspx

Cheers
Joe

Hugo · April 2012

Hi Joe

Appreciate the info, very helpful - of course the filtering is a non-issue actually - we are using LINQ after all so can render whatever we need, just been a while since I even looked at LINQ and the driver is my main focus.

Now - after some small tweaks (again this is experimental/feasibility stuff here) I am getting this when I try to run a query:

The type or namespace name 'TypedDataContext' does not exist in the namespace 'LINQPad.User'

I have populated the tree view using the metadata in our data source, and enabled the context menu etc.

The node I am testing with is named Watchitems - the IEnumerable member in the data context class is named Watchitems and the generic T is Watchitem.

This is a contrived example/test as I explore the whole driver concept - but I was getting execution when the tree view was based on the Astoria stuff - but now that the tree view is being created for our data we get this runtime error.

Just to explain I have taken the Astoria code sample - which works - and incrementally tweaked it to gain insights/understanding. The version that 'worked' simply had a modified property in the dcDataService class that used our T and returned our data of type T.

I commented out the src code gen stuff and made it a fixed src file so that I could do these things, once all is stable I will write my own DataService class generator - if I need to (it seems I do).

Clearly something happens when a node is clicked and a query runs, and the error "The type or namespace name 'TypedDataContext' does not exist in the namespace 'LINQPad.User' " means something but I have no idea, nor are any exceptions raised under debug.

Any help much appreciated!

Hugo

Hugo · April 2012

I'd also like to clarify something - DataContext (it seems) is intimately associated with SQL and LINQ - but what if we are not accessing a relational data source at all? what if we are accessing plain old IEnumerable[T] collections?

For the time being forget about how these come to be, assume the driver can peform some data load operation and immediately "see" IEnumerable[T] collections, assume the data store exposes these without any need to work with SQL, O/R mappings or anything like a DataContext.

Does the driver model support this scenario or is it currently expecting to work only with DataContext and SQL like data sources? I'm new to WCF, DataContext and all this - so far as I am understanding things, LINQ too has no direct association with these itself.

I can (if neccesary) contrive a dynamically generated data context class myself but there is no service url or anything in our case so I am unsure of how to force this DataContext model to work with pure in-memory collections.

Thanks

JoeAlbahari · April 2012

TypedDataContext is the default name of the typed data context that LINQPad expects you to create with a dynamic driver. In other words, in the GetSchemaAndBuildAssembly method, you must create an assembly that contains a class called "TypedDataContext" with a public constructor, in the namespace "LINQPad.User". LINQPad passes these two values into the typeName and nameSpace parameters of the GetSchemaAndBuildAssembly method (so don't hard-code them). And because these are ref parameters, you can change their values if you prefer. A reason for changing the class name is if you're working with a third-party code generator that doesn't let you specify the class name yourself.

LINQPad's notion of a "typed data context" is any non-abstract class that has properties or fields or methods that the user can query. It does not have to be based on LINQ to SQL's DataContext class, or anything else for that matter. The properties/fields will typically be of type IEnumerable or IQueryable (so the user can run LINQ queries over them), but they don't have to be. For example, you might emit an assembly with the following class:


public class TypedDataContext
{
  public IEnumerable<string> CustomerNames { get; set; }
  public IEnumerable<int> Bar { get; set; }
  public void DoSomething() { }
}

When the user runs a query, LINQPad will subclass this and instantiate the subclass. For instance, suppose the user types this and hits F5:

CustomerNames.Dump();
Bar.Where (b => b > 10).Dump();
DoSomething();

LINQPad will convert this into the following code, compile it, and then run it:

public class UserQuery : MyTypedDataContext
{
  public void RunUserAuthoredQuery()
  {
    CustomerNames.Dump();
    Bar.Where (b => b > 10).Dump();
    DoSomething();
  }
}

Hugo · April 2012

That was most helpful Joe - got a basic version running here now and have a much clearer idea of how to go forward and package this all up and create an initial beta driver for Persistore.

Another question - and I can live with this for time being I think - our datapools are hierarchical - and data is stored in a directory hierarchy, this is quite unlike SQL Server where there is a conceptual level of "tables" in which all data sets reside.

So by way of example, at the root of a datapool there may be two dirs "current_data" and "old_data" and we may have collections of type T in each perhaps named "Customers".

So - again conceptually - we could refer to TWO data collections of the same type "current_data.Customers" and "old_data.Customers".

I may be wrong but this seems to suggest the LINQPad driver model can't work with this as it has a single TypedDataContext class - yet we need two one for collections within "current_data" and one for collections within "old_data".

This would seem to suggest some kind of TypeDataContext that might "look like" this:


public class TypedDataContext
{  
public IEnumerable<string> CustomerNames (string path);
public IEnumerable<int> Bar (string path);
public void DoSomething() { }
}

If this were supported then LINQPad could do as it does now and treat properties as it dos now, but if a clicked name (in this example 'CustomerNames') is found to map to a method not a property - then it could pass the entire tree-view path of the clicked node.

In this case our own logic could then use the path text to ascertain which actual directory we need to query, this is of course just an idea - you may have a way already or may have better ideas for supporting such a hierarchical store.

Thanks again - and back to my beta....

JoeAlbahari · April 2012

I don't understand what you're suggesting: the example you posted matches what was in my previous example.

You can re-use the same type name by giving it a prefix or putting it in a different namespace, or putting it in a nested class. To access the "old" data, you could write a separate typed data context and expose it like this:

public class TypedDataContext
{
  public OldDataContext Old;
  ...
}

Take a look at how LINQPad handles multiple databases in a single query - it's a similar situation. Create a connection to SQL Server and follow these instructions:
http://www.linqpad.net/FAQ.aspx#cross-database

You can see how it works underneath with .NET Reflector. Press Shift+F1 to reflect the current type/member.

Hugo · April 2012

Thanks Joe - much appreciated - I will indeed examine these suggestions. (My example was just a way of suggesting methods rather than properties as being exposed by the context).

A nested class would be an excellent way of doing this - let me explore these ideas now that you have explained this to me.

Thx

Hugo

Hugo · April 2012

OK this looks fine - I just created simple nested wrapper classes and I can now support the hierarchies. The name of the node that is now clicked (the enumerable collection) is a fully qualified string that maps exactly to the full name of the member in the TypedDataContext and its nested container class.

A suggestion emerges here too:- what about an ExplorerItem having a "DisplayedName" property - this could default to the current 'Name" property. But if a driver sets this string property then this becomes the text displayed as the node's name.

This will allow a node to have a simple (unqualified) name displayed in the tree - but have it's fully qualified name available in the Name property - which is the name used by LINQPad to resolve the eventual member collection property.

Anyway this was the last issue I need to resolve - I think I can create a solid driver now.

Thanks for help.

Hugo

Hugo · April 2012

This is now looking very good - I created a metadata wrapper generator algorithm, able to gen the data context class and as many nested internal classes as are needed to represent the directory hierarchies in the datapool.

I can now connect to any datapool and run queries against any data in it, no matter where that data appears in the overall hierachy.

Just need to do a tidy up and a set of formal tests and we're good.

Thanks for your help with all this Joe.

Hugo

Writing a driver for a non ADO data source.

Comments