Feedback needed: command-line interface

JoeAlbahari

I'm currently considering how to enhance LINQPad to enable a command-line interface.

What I need right now is an idea of exactly how people would use this feature - what are the use cases?

In particular, I need to consider how to deal with output, which is currently formatted as HTML and displayed in an embedded web browser control (or in data grids). Should this be formatted as plain text instead and directed to the console output?

For queries that require a connection, should LINQPad requery the schema and rebuild the typed data context each time you call the command-line executable (as it does currently with the GUI), or should LINQPad somehow cache the typed data context to avoid the overhead?

And how should exceptions be handled?

Thanks!
Joe

Edit: If LINQPad were to cache the typed data context, the cached copy would be used only if the schema hadn't changed since the last run.

apuddefoot

Hi Joe,

For my needs, this feature would be great. I use Linqpad mostly as a c# scratch pad for updating and collecting data from Domain Controllers. linqpad is great because I don't need to have hundreds of tiny VS solutions all over the place.

When I perform mass updates to Active Directory the input data generally comes from csv's and any output/status is usually sent to a csv (i create my own) or plain text. I may use the output window for status of where the process is up to but any required output goes to csv. So in my case the format of the linqpad output is not relevant. However console output would be good for long running operations.

I would prefer that the schema be re-queried as the connections I run operations on do change often and would not want to risk connecting to a cached connection.

Exceptions should be written to output and the console.

I hope this helps. Keep up the awesome work!.

Andrew Puddefoot.

Olaf

As Andrew mentioned: linqpad scripts replace a lot of clumsier VS projects.

As for your questions:
* output to console as text will be sufficient for me. Make it the standard and possibly add html rendering as an option
* re-query the schema on every execution of the script
* exception should be written to the STDERR, not the standard output

Olaf

markheath

Here's a few of my use cases:

1. I have lots of scripts that check a large visual studio solution for various problems. I want our build machine to run them as part of a daily build. Exceptions/errors would need to be written to stderr so msbuild can handle it properly.

2. I have a number of backup tasks written as LINQPad scripts, which I launch manually at the moment. A command line interface would make scheduling these backups easier.

3. I use the NuGet integration to write lots of simple utilities for my NAudio library. For example I wrote one for a colleague yesterday to convert from raw IMBE to WAV. If scripts like that could also be easily turned into command line tools, it would be brilliant.

thanks again for LINQPad. It's one of my favourite utilities ever. Looking forward to seeing what you do with this feature.

Valentin

Hi Joe!
Most often LinqPad usage scenario for me are tons of daily/weekly/monthly/etc well-formatted reports.
* It would be great if I could redirect the formatted output into a file via command line options, usage example:

linqpad.exe "c:\script.linq" /execute /out:report.html

And I'm not too interested in the console output, except debug traces maybe.

* Schema caching is a good idea, it could be optional (switched by a command line parameter)
* exception output into STDERR is quite enough for me.

mortenstr

Hello Joe,

As mentioned by the previous posts here, for me and the tasks I want to perform it is good enough to route output to console and errors to the STDERR.

Thanks!

Tormod

I hope it handles query dependencies to binaries, data sources and nuget packages. I also hope that it accepts a search path parameter for unresolved dependencies and as a starting point for binary dependencies given as relative paths.
I would like to pass parameters to the query from the cmd line. Not quite sure how it would work. One example would be to demand that the query type be "C# Program" and then use regular "params" keyword. But then how would you populate values when using F5 to run it....? Another possibility is to have parameters as query properties and accessible through name/value pair which can be accessed through the "Util." class.
Another challenge is how the various output/export csv/html formatting/non-formatting options translate to the automated run. I also hope that even if the html output is seven levels deep, that the html output could be "pre expanded" to only three levels.
Are you all just thinking CMD or are you considering powershell.

ZachBuckner

My use case is identical to Valentin's. Reports, and outputting to HTML would be great. I use my own DataContext .DLL.

frankiansmith

We use LINQPad as a scripting environment for an in-house application that operates on SQLite databases. When a user runs the application, she could be running one of many available versions of the application, using one of many SQLite databases.

We have a repository of LINQpad scripts. Currently, when our application invokes one of these, it manually patches the XML in the script with the locations of the proper assemblies, and manually patches the value of a magic string variable with the path to the database.

LINQPad is invoked by our application, so at that point we know exactly which assemblies to use and which database to use, so if I had a way to pass that information to LINQPad and the running script that would be wonderful. Perhaps command-line options to specify references and namespaces to add, and a way to pass arguments to the script?

CPfohl

First of all: AWESOME and Thank You! I feel like a lot of people have been using LinqPad similar to this already, making it official, and text-prompt ready is fantastic.

Like other people have been saying, I've used LinqPad where a full-on VS project would be way heavy-handed. It replaced python for me as a quick scripting tool. What was especially helpful was the ability to Dump() and explore an object. That, however, is simply during development. After I've finished developing all I'd really need was a way to run it from the command prompt. I wouldn't want to replace LinqPad as an IDE (it does its job very very well).

In terms of output: I think you might want to consider adding a formatter option to Dump(). Perhaps even a `FormattedDump()`. Then you don't have to make the choice for people. The default could be to spit out the object as a CSV.

I like how LinqPad currently handles connections. I wouldn't want things cached.

Exceptions should be Dumped with a full stack available by default (or cmd-line option?) Add an appropriate return code and stop running.

Regardless of what you choose, I'm sure it will be fantastic. If the cmd version is even half as useful as the GUI version you'd have a pretty darn amazing product.

Chris

PascalLaurin

Pretty much what Valentin said.

I did wrote my own command line interface for LinqPad (but it's really buggy and unstable) and also had a few other options:
/show to open the outputted html file in the default browser immediately
/smtp and /sento to send the outputted html file via email (great for reports with scheduled tasks)

Finally I'm allowing to add a Main(params string[] args) method in the query and provide extra parameters when calling the command line. When running the query inside LinqPad in the Main() method I can call Main(params string[] args) with the list of parameters I want to test the query with.

I don't know if all of this could be useful to others.

Anders

I agree with PascalLaurin on every point.
I also agree that exceptions should be directed to STDERR.

brownreidl

I use LinqPad much as the other posters -- as an alternative to having tens to hundreds of small (unmanageable) VS projects.

For us, LinqPad remains resident 24 hours a day on a production machine. There is at least one script that is always running that moves files, does data validation, and sends emails if errors are detected, all on a 24 hour schedule.

Other scripts (20-40 of them) provide file decryption and file manipulation (data validation, directory management, etc.) services that do not involve a database. These are currently loaded into a LinqPad tab by an operator, one or more variable values might be changed in the script representing parameters needed by the script for that run, and then the script is run by the operator. The output window is only used for run status messages, which are normally text. (The real output of these scripts is either modified files, or files moved to new locations, etc.)

Another 10-20 scripts are tied to various databases and provide reports. Output is viewed as html in the output window.

Therefore, my need meshes fairly closely with Valentin's command line format, but I would also like to be able to pass parameters as suggested by PascalLaurin with a "Main()" method. Perhaps something like:

linqpad.exe "c:\script.linq" /execute /out:report.html /param:"baa ram ewe"

As for some of the other issues:

* I would not like to see schemas cached by default. Ours change too frequently.
* Error output to STDERR would be fine. Stack trace would be nice.
* Results window output can be either text or HTML, and it should go to STDOUT if no option is given to send it to a file (e.g. Valantin's "/out:" override). For my reports, it would be nice if there were an option to get HTML output if text is the default.
* References should be resolved anew for each command line invocation.

When this feature is available, I can envision running several of these scripts from the task scheduler. Many of the file maintenance scripts will become command line utilities that take command line arguments -- a much simple interface for our operations staff.

I am really excited that you are now considering adding this functionality. The ability to invoke LinqPad scripts from the command line is the only feature that has prevented LinqPad from becoming the primary scripting language in our environment. (I look forward to writing Powershell scripts that consist of a single line invoking linqpad.exe with a C# script to do the real work.)

JohannesF

in short
Errors ouput to STDERR
Output from script to STDOUT (so it can be used for pipes)

My use case would be test pad and for scripting (instead of dos batch or similar).
For this it would be great if LinqPad would work together with Powershell (also for piping in/out)

paul_petho

Hi, thank you for this great feature in advance.
I'm sure your tool could compete with powershell...
My need is something similar to Valentin's, and brownreidl's param tag:
linqpad.exe "c:\script.linq" /execute /out:report.html /param:"baa ram ewe"

If feasible it would be very nice, if one main linq "script" can call another linq scripts. This way LINQPad can cache the typed data context in a batch environment to avoid the overhead when executing several scripts running in the same or precisely in similar contexts/environments...
Another way of achiving this caching would be if LINQPad can enter into a command line mode (as if you were in a REPL cycle, but only for invoking linq scripts):
linqpad.exe /mode:cmd
script1.linq
script2.linq...
bye

Maybe mailto swith could also be very useful.
Thanks, Paul...

kswaroop

Hi,
Coming from a UNIX background, I would like to see the cmd line version have similar behaviour so that output from one cmd can be piped to another etc (also already available in cmd window and powershell).
So I would vote for some of the suggestions already made above, plus some more:
- to go in cmd line mode use specific option eg /file:script.linq
- output to stdout (no need to specify /out option, one could simply redirect output to a file using '>' on command line)
- errors to stderr
- a way to pass on any options and args not consumed by linqpad to the script itself (no need to specify /param option, just make those available to script as equivalent of string[] args to main, there is already concept of some reserved words in linqpad scripts, eg static UserQuery ... another named __ARGV__ etc won't hurt)
- optionally progress to stderr (similar to line highlighting in ide mode)
- Dump format options
. plain text: take field separator, and a record separator as options:
. html
. options around nesting, either denormalize/flatten or indented/leave fields empty to get break on next change, eg for plain text option:
FLAT:
a1,b11,c111
a1,b11,c112
a1,b11,c113
a1,b12,c121
a1,b12,c122
a2,b21,c211
IDENTED:
a1,b11,c111
,,c112
,,c113
,b12,c121
,,c122
a2,b21,c211
- please also provide unix style method of specifying options, something like
linepad -f script.linq -o=text -d'|' -r "%\n" -nest=indent
would produce (taking same example as above)
a1|b11|c111%
||c112%
||c113%
|b12|c121%
||c122%
a2|b21|c211%
- REPL mode (great idea)


Thanks

ehoffmann

Output:
I believe that I'd normally want to manage my console output explicitly, via calls to Console.Write Console.Error, etc..

I don't think I'd normally want/expect the results of queries, .Dump(), Util.Image, Util.RawHtml, etc to go to the console... those are explicitly for the results window/interactive usage..

If I could register a hook for those calls in the script that would let me format the output, that would be awesome:
Util.RegisterDumpHandler((objectBeingDumped) => { /* my formatting/output.. to console.. to disk..etc */ });


If the HTML content was needed, perhaps a command line flag indicating where to dump the HTML file/any generated images.

Schema rebuild
Rebuild each time, just like the GUI

Command line arguments
Woud be a thing of beauty to be able to access the command line parameters, either from
- Util.GetCommandLineArgs()
- or magically getting passed in the void Main(string [] args) when the script is setup as a 'program'
- or just via System.Environment.GetCommandLineArgs[]

Being able to load/run other scripts in my script
Would be interesting, though not a key point for me.

Being able to specify which data context (on command line)
Would be awesome

Being able to change it programmatically inside the script
Would be even better (loop & report over N different DBs..)

Hogan

My two cents :

Output
Output to an HTML if Dump() is used but also text output to stdout and stderr if Console.Write is used. This would enable scheduler / service use.

Connections
Some way to control connections would be very useful.

Parameters
Would be quite useful.

Input
Reading from standard in could replace the need for command line parameters (both would be nice too.)

rekna

My main usage would be as a replacement for cmd or even powershell (as a programmer , powershell's syntax is just to far away from programming).
Commandline arguments for the scripts to be executed are a must. I would also suggest being able to choose between running linqpad with the user interface or only as a console. Of course when running in console mode, output is more limited. In console mode the Dump method as far as I'm concerned should be ignored (I would like to be able to leave the Dump method inside scripts, but since html output in console is not possible, it should output nothing).
The location of the script, using the Utils class is a must as are relative paths in the reference dialog.

ivanakcheurov

As many people have mentioned, we'd like to use LinqPad as a scripting tool for command line. Perhaps it would replace cmd/poweshell/fsx scripts.

Please add easy inbuilt means of launching another applications from LinqPad scripts and capturing/processing their output.
Then it could really replace cmd/powershell scripting. An example:
Shell.Start("your.exe", "/switch1", "param2", HandleOutputMethodOrLambda).

The main point is to avoid linking quirky self-made libraries and to have clear, simple and powerfull API for that. Easy things must be easy, complex ones should be possible.

ulrich

It would be really cool if this new command line runner would support multiple platforms (using mono) => no P/Invoke.
... or at least partially (no Linq2SQL, ...)

JuergenB

Hi Joe
- To have the errors on STDERR would be good
- Output in HTML in addition to plain text would be great
- If possible make the existing export options (Word / Word-Formatted / Excel / Excel-Formatte / HTML) available to the command line or in the script itself.
linqpad /script /outputformat:word-formatted /outputFile test.docx
- If it would be possible to use Util.SaveResults(“c:\temp\dummy.html”, ResultFormat.HTML) in a script would be great.
- Having the option to add a dictionary of named input values would be great, maybe something like this linqpad.exe /script: /execute /param:name=test /param: user=testuser, to access the parameters in an IDictionary Util.Parameters
- An option to select the connection to run against would be great e.g. /useconnection:, this would be helpful because we use the exact same queries against different data sources on different machines.
Thank you again for that great tool, which is one of the best things ever invented in the .NET-space.

andremoraes

joe, any idea if basic cmd line support will be added in the new beta?
just really looking for
linqpad.exe myscript.linq "param1", "param2", "param3", ....

Brian

Yup being able to execute a LINQ query via PowerShell and dump it to a csv would be perfect. Then I can just Import-CSV that info in PoSh.

linqpad.exe "c:\script.linq" /execute /out:report.csv

jsytniak

One use case would be to produce adhoc/short lifespan reports. Kick off a script with a couple of command line switches... one for arguments for input parameters to the the script (values that I might now collect interactively via Util.ReadLine)... and then another switch for what to do with the output. Something like...

linqpad.exe "c:\script.linq" /execute /args:"a,b,c" /outcsv:report.csv

-or-

linqpad.exe "c:\script.linq" /execute /args:"a,b,c" /outhtml:report.html

jsytniak

I would dump stderr to the console.... if one wishes they can pipe it to a file.

jamesmanning

Not sure where this has ended up, but as much as I hate scope creep, looking through the thread, I think it might make the most sense to create an object model (as simple as possible).

With that available, people could use it from C# or PowerShell and get a 'full fidelity' object graph as output (or whatever data structure(s) you'd want to be able to offer back - maybe even something 'complete' like everything that the LINQPad code already keeps track of during the execution?)

The cmdline becomes the only place that is forced into consuming things in a serialized format (although you could certainly use xml or json or whatever and then pull it back into an object in the calling script/code/whatever).

An object model / API that lets consumers get back the real objects (or something higher fidelity than serialized html) can be leveraged for people that want xhtml or csv, since LINQPad already has all the heavy lifting for serializing out object graphs into xhtml and csv.

It also enables more and better integration scenarios - as a random example, with the relatively recent support for running C# and PowerShell within a node.js application, a LINQPad object model would allow some useful scenarios there (LINQPad as swiss-army-knife-data-and-anything-else engine, letting others just leverage all its work and present/integration/etc).

I realize that getting into the mode of supporting a public API surface is likely not very enticing, so IMHO for your own sanity (Joe), I think such an object model could definitely be treated as unsupported/experimental/best-effort/etc - as long as what it offers is complete enough (at least in terms of getting data out of a completed query execution), then it's reasonable to let the community deal with the creation and maintenance of a LINQPadContrib NuGet package (or similar) that others could build on as a base for their integration work.

Thanks, Joe!!

ksecrist

To me this would be very powerful, for all sorts of things:

From command line, run linqpad and provide .linq "program" file.

Have that script run with the script's last saved database properties.

I could take time to port things to Visual Studio, but honestly for the things I'm thinking of, misc random data extracts, batches, etc - there isn't an actual reason to need to turn them into Visual Studio exe's if I could have this feature.

Thanks ++ a million.
-K

Nate630

Here is the use case I am doing right now - dumping a SharePoint list. It launches the LINQPad main window, however.

LINQPad.exe "Queries\DumpNetworkContacts.linq" -run

DumpNetworkContacts.linq looks like this....

var x = Contacts.ToList();
//Removed: data transform on Contacts data
Util.WriteCsv(x, @C:\contacts.csv);

var linq = Process.GetProcessesByName("LINQPad").FirstOrDefault ();
linq.CloseMainWindow();


It would be nice to just say '-run -quiet' to suppress the main window and allow LINQPad to handle the exit after the script has completed.


Nate

zinovate

I could use this BigTime

JoeAlbahari

A LINQPad command-line beta is now available:

http://www.linqpad.net/beta.aspx
http://www.linqpad.net/lprun.aspx

Please let me know your feedback.