JSON output of lprun8 does not work properly for some cases for DumpContainer, Hyperlinq and Word

Also, the following code when run with lprun8:

    new
    {
        Foo = Util.HorizontalRun(withGaps: true, ["Exception", Enumerable.Range(1, 3).OnDemand("NOT displayed")])
    }.Dump();

throws a JsonSerializationException with message Self referencing loop detected for property 'dc' with type 'LINQPad.DumpContainer'. Path 'Foo.Elements[1].Content.Action.Target'.:

JsonSerializationException: Self referencing loop detected for property 'dc' with type 'LINQPad.DumpContainer'. Path 'Foo.Elements[1].Content.Action.Target'.
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.CheckForCircularReference(JsonWriter writer, Object value, JsonProperty property, JsonContract contract, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.CalculatePropertyValues(JsonWriter writer, Object value, JsonContainerContract contract, JsonProperty member, JsonProperty property, JsonContract& memberContract, Object& memberValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeList(JsonWriter writer, IEnumerable values, JsonArrayContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeISerializable(JsonWriter writer, ISerializable value, JsonISerializableContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Serialize(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.JsonConvert.SerializeObjectInternal(Object value, Type type, JsonSerializer jsonSerializer)
   at Newtonsoft.Json.JsonConvert.SerializeObject(Object value, Type type, Formatting formatting, JsonSerializerSettings settings)
   at Newtonsoft.Json.JsonConvert.SerializeObject(Object value, Formatting formatting, JsonSerializerSettings settings)
   at LINQPad.ObjectGraph.Formatters.TextOutputWriter.NodeToText(ObjectNode node)
   at LINQPad.ObjectGraph.Formatters.TextOutputWriter.FormatObject(Object value, String heading, String lastHeading, GraphOptions options, Boolean& inline, TimeSpan& formattingTime)
   at LINQPad.ObjectGraph.Formatters.OutputWriter.WriteObject(Object value, String heading, GraphOptions options, Boolean noNewLine)
   at LINQPad.ObjectGraph.Formatters.OutputWriter.WriteLineObject(Object value, String heading, GraphOptions options, Boolean noNewLine)
   at LINQPad.Extensions.DumpCore[T](T o, String description, Nullable`1 depth, Nullable`1 collapseTo, Boolean toDataGrid, String exclude, String include, Nullable`1 alpha, Boolean noTotals, Nullable`1 repeatHeadersAt, Boolean includePrivate, Nullable`1 maxRows, Nullable`1 useInvariantCultureForNumbers, DumpFormatStrings formatStrings, Boolean forceObjectExpansion, Boolean descriptionWasAutoGenerated)
   at LINQPad.Extensions.Dump[T](T o, String description, Nullable`1 depth, Nullable`1 collapseTo, Boolean toDataGrid, String exclude, String include, Nullable`1 alpha, Boolean noTotals, Nullable`1 repeatHeadersAt, Boolean includePrivate)
   at LINQPad.Extensions.Dump[T](T o)
   at UserQuery.Main() in C:\Users\DavidNemeti\AppData\Local\Temp\LINQPad8\__hglzagvf\dzauzp\LINQPadQuery:line 38
   at LINQPad.ExecutionModel.ClrQueryRunner.Run()
   at LINQPad.ExecutionModel.Server.RunQuery(QueryRunner runner)

I am aware of that LPRun is not about user interactivity; however, it displays (by default) a JSON representation of the dumped objects. LPRun should also display the text of OnDemand, and most of all, it should not throw any exceptions.

The following code displays the text on the UI, and allows user interactivity. It should also display the text in LPRun's output (of course, in this case there is no user interactivity).

Util.OnDemand("This text should be displayed both on the UI and in LPRun's output", () => 5).Dump();

I run my LINQPad script both in LINQPad UI and by LPRun.

It shows information which can be quickly calculated, and I want this information to be displayed both in LINQPad UI and in LPRun output.

Furthermore, if I run the script in LINQPad UI, I want to be able to click on the displayed information in order to display a more detailed, slowly calculated view.

Something like this:

var numbers = Enumerable.Range(1, 5).ToArray();

numbers
    .Select(number => Slow(number))
    .OnDemand(numbers.Length.ToString())
    .Dump();

int Slow(int number)
{
    Thread.Sleep(500);
    return number * 2;
}

Unfortunately, it does not display the description of OnDemand in LPRun's output.

So I tried to work around this problem (hoping that eventually the original OnDemand will work as well):

numbers
    .Select(number => Slow(number))
    .OnDemandWorkaround(numbers.Length.ToString())
    .Dump();

I tried this workaround, which is a little bit ugly, but it throws the JsonSerializationException that I mentioned earlier:

    public static object OnDemandWorkaround<T>(this IEnumerable<T> items, string description) =>
        // NOTE: this workaround also fails in lprun8
        Util.HorizontalRun(withGaps: true, [description, items.OnDemand("[...]")]);

The following workaround works in LPRun, but its output is extremely ugly (in LINQPad UI the description is shown together with the [...] text, and in LPRun's output the description is shown and OnDemand is omitted, but at least I get no exceptions):

    public static object OnDemandWorkaround<T>(this IEnumerable<T> items, string description) =>
        // this workaround works, but it is ugly
        new
        {
            Value = description,
            Calculate = items.OnDemand("[...]")
        };

Yes, I know the CMD symbol feature. However, it is not a proper solution for me because of the following reasons.

First, I would still need to use the OnDemandWorkaround method (although with a better implementation), and pay attention to always use it instead of OnDemand. This is error-prone, because if I accidentally use OnDemand somewhere, LPRun will fail.

public static object OnDemandWorkaround<T>(this IEnumerable<T> items, string description) =>
#if CMD
    description;
#else
    items.OnDemand(description);
#endif

Second, I need to create and save a textual version of some of the important dumped output both when running in LINQPad UI and in LPRun.

TextWriter MessageWriter = (TextWriter)Activator.CreateInstance(
            "LINQPad.Runtime",
            "LINQPad.ObjectGraph.Formatters.TextOutputWriter",
            ignoreCase: false,
            BindingFlags.Public | BindingFlags.Instance,
            binder: null,
            args: [QueryResultFormat.Text],
            culture: null,
            activationAttributes: null
            )!
            .Unwrap()!

Unfortunately, TextOutputWriter is internal, so I need to use reflection here. I would prefer to use e.g. Util.CreateTextOutputWriter(), similar to the existing Util.CreateXhtmlWriter().

When I call DumpImportant then it will dump to the output (either LINQPad UI or LPRun output) and to the MessageWriter which will be saved later.

    public static T DumpImportant<T>(this T value, string? description = null)
    {
        if (description is not null)
        {
            value.Dump(description);

            MessageWriter.WriteLine(description);
            value.DumpTo(MessageWriter);
        }
        else
        {
            value.Dump();
            value.DumpTo(MessageWriter);
        }

        return value;
    }

    public static T DumpTo<T>(this T value, TextWriter textwriter)
    {
        textwriter.WriteLine(value);
        return value;
    }

Thus, my problem is that TextOutputWriter (which is used both by me directly and by LPRun to render its output) does not handle OnDemand properly.

One might think that I could use my own JSON serialization logic to save my important output, but I do not want to create a textual serializer from scratch to handle all of the cases (WordRun, VerticalRun, Hyperlinq, etc.) especially when TextOutputWriter already exists, it just would need to be fixed. And even if I used my own JSON serialization logic for save, I would still need to use OnDemandWorkaround instead of OnDemand because LPRun demands it in order to work around the faulty operation of TextOutputWriter.

In summary: LPRun should display the same output in textual form as LINQPad UI displays in HTML form (except images, of course); and what is even worse, LPRun currently fails with exception for some cases which is clearly a bug.

Thanks, but I know the behavior of Lazy and OnDemand in LINQPad very well.

Just fix the errors and move on. I wonder why you are so in desperate need for OnDemand?

I just have told you in great detail: the need to avoid OnDemand everywhere in the code, and using OnDemandWorkaround or some other workaround is clumsy, fragile and error-prone. I need to avoid it now, and I need to remember every time I change or extend the code to avoid it. And not just I need to avoid it, but my coworkers as well who are also editing the code.

An how LPRun should render Lazy and OnDemand which requires user interaction and might be expensive to evaluate (or side-effecting)?

I have also elaborated on this: OnDemand may not just show a "Click me" text (which has purpose only when interaction is possible), but it may show a brief information which can be quickly calculated, and I want this information to be displayed both in LINQPad UI and in LPRun output.

Of course, if I run the script in LINQPad UI, I want to be able to click on the displayed information in order to display a more detailed, slowly calculated view. LINQPad behaves so, which is good.

Of course, in LPRun output I cannot click on the displayed brief information, so there is no way and no need to initiate the slow calculation which would display a more detailed.

What I want is very simple and intuitive: I would like to see the same output in LPRun output as I see in LINQPad UI initially without any interactions.

And speaking of Lazy: LPRun currently displays Lazy in a very wrong way by executing the slow operation and displaying an internal view of the Lazy object where you can see that although IsValueCreated is false, yet the Value is calculated (but this is another, although connected topic beside the OnDemand topic).

The following script:

new Lazy<int>(() => 1 + 2).Dump();
Util.OnDemand<int>("This is what I would like to see in the output, because it may contain real information, not just a \"Click me\" text.", () => 4 + 5).Dump();

currently results the following LPRun output:

{
  "IsValueCreated": false,
  "Value": 3
}

while the following LPRun output would be desired:

Lazy<Int32>
This is what I would like to see in the output, because it may contain real information, not just a "Click me" text.

or LPRun can omit Lazy objects altogether (in this case I can let go of the "same output" principal, because in LPRun output a displayed Lazy<Int32> text has no important information, and it has no purpose since it cannot be clicked):

This is what I would like to see in the output, because it may contain real information, not just a "Click me" text.

E.g. the following script:

var numbers = Enumerable.Range(1, 5).ToArray();

numbers
    .Select(number => Slow(number))
    .OnDemand(numbers.Length.ToString())
    .Dump();

int Slow(int number)
{
    Thread.Sleep(500);
    return number * 2;
}

should result the following LPRun output:

5

And what's the point to use in in LPRun instead of LINQPad? Just save results as JSON, etc.

My scripted can be executed in different scenarios:
- locally on our developers' machine (in which case I want to provide a much richer, interactive UI experience, hence the usage of LINQPad UI)
- on our build server (in which case I need to use LPRun, and I need to create a text output so it can be easily read in the build server log)

Thanks, but I believe that this code is a little bit intrusive as it defines its own Lazy<T> type. It is not a good idea on its own, let alone when I am dumping objects from other assemblies which use .NET's Lazy<T> type instead of this special Lazy<T> type, or when someone uses Lazy<T> type by its fully qualified name System.Lazy<T>.

BTW the code outputs this in LPRun:

Enumerable: [
  2,
  4,
  6,
  8,
  10
]
Util: 9
Util: A B C
Lazy: Executed
Lazy: 1

but this would be the desired output:

Enumerable: 5
Util: This is what I would like to see in the output, because it may contain real information, not just a "Click me" text.
Util: A B C
Lazy: Lazy`1

Of course, this can be fixed:

public static class Extensions
{
    public static string OnDemand<T>(this IEnumerable<T> e, string s) => s;
}

static partial class Util
{
    public static string OnDemand<T>(string s, Func<T> r) => s;

    public static object HorizontalRun(bool b, params object[] p) =>
        global::LINQPad.Util.HorizontalRun(true, p);
}

static object? ToDump(object? input)
{
    if (input?.GetType().GetGenericTypeDefinition() == typeof(Lazy<>))
        return $"Lazy<{input.GetType().GetGenericArguments().Single().Name}>";
    else
        return input;
}

There is no need to define a special Lazy<T> type.

BTW, there remained still a lot of problems with the LPRun output that I have not talked about. E.g. HorizontalRun and WordRun is not displayed nicely as text in a row when LINQPad.ObjectGraph.IMetaGraphNodeGen.InLine is true, rather it is displayed in an ugly JSON serialized from. Or e.g. Util.Highlight("world") is not displayed as world, rather in an ugly JSON serialized form of the internal data structure:

  "F": {
    "Data": "world",
    "Class": "highlight",
    "Style": null,
    "IsInline": null
  }

But in the meantime I have managed to work around all of these problems simply by defining a static ToDump method, without redefining any .NET or LINQPad types, so now I am getting the desired output in LPRun. Unfortunately, the LINQPad structures I needed to use are internal in the LINQPad.ObjectGraph namespace, so I had to use reflection to access them.

But my main problem is that LPRun should have the desired output without any workarounds, so my only goal here is to report a bug ticket and wait for @JoeAlbahari to fix these problems.

However, the problems (both the exception and the style problems) can be work around by using LINQPad's powerful ToDump feature. So by using the following code:

static object? ToDump(object? input) =>
    TextDumpWorkaround.ToDumpWorkaround(input);

public static class TextDumpWorkaround
{
    // Workaround is needed until the following bug is fixed in LINQPad:
    // https://forum.linqpad.net/discussion/3335/json-output-of-lprun8-does-not-work-properly-for-some-cases-for-dumpcontainer-hyperlinq-and-word
    public static object? ToDumpWorkaround(object? input)
    {
        if (input is null)
            return null;

#if !CMD
        return input;
#endif

        return input switch
        {
            Hyperlinq hyperlinq =>
                hyperlinq.Text,
            DumpContainer dc =>
                dc.Content is Hyperlinq hyperlinq ? hyperlinq.Text : ToDumpWorkaround(dc.Content),
            { } when (IsHeadingPresenter(input, out var heading, out var content)) =>
                ToDumpWorkaround(Util.VerticalRun(heading, content)),
            { } when (IsISingleMetaGraphNodeGen(input, out var content)) =>
                ToDumpWorkaround(content),
            { } when (IsIMultiMetaGraphNodeGen(input, out var content) && content is not null) =>
                IsInline(input)
                    ? string.Join(IsWithGaps(input) ? " " : string.Empty, content.Select(ToDumpText))
                    : content.Select(ToDumpWorkaround),
            _ => input
        };
    }

    private static string? ToDumpText(object? value) =>
        ToDumpWorkaround(value)?.ToString();

    private static bool IsISingleMetaGraphNodeGen(object obj, out object? content)
    {
        var interfaceType = obj.GetType().GetInterface("LINQPad.ObjectGraph.ISingleMetaGraphNodeGen");

        if (interfaceType is not null)
        {
            content = interfaceType.GetMethod("GetContent")!.Invoke(obj, []);
            return true;
        }
        else
        {
            content = null;
            return false;
        }
    }

    private static bool IsIMultiMetaGraphNodeGen(object obj, out object?[]? content)
    {
        var interfaceType = obj.GetType().GetInterface("LINQPad.ObjectGraph.IMultiMetaGraphNodeGen");

        if (interfaceType is not null)
        {
            content = (object?[]?)interfaceType.GetMethod("GetContent")!.Invoke(obj, [])!;
            return true;
        }
        else
        {
            content = null;
            return false;
        }
    }

    private static bool IsWithGaps(object obj)
    {
        var type = obj.GetType();

        return type.FullName is "LINQPad.ObjectGraph.HorizontalRun" or "LINQPad.ObjectGraph.WordRun"
            ? (bool)type.GetField("WithGaps")!.GetValue(obj)!
            : true;
    }

    private static bool IsHeadingPresenter(object obj, out string? heading, out object? content)
    {
        var type = obj.GetType();
        if (type.FullName == "LINQPad.ObjectGraph.HeadingPresenter")
        {
            heading = (string?)type.GetField("Heading")!.GetValue(obj);
            content = type.GetField("Content")!.GetValue(obj);
            return true;
        }
        else
        {
            heading = null;
            content = null;
            return false;
        }
    }

    private static bool IsInline(object? obj)
    {
        if (obj is null)
            return true;

        if (!TryGetIsInline(obj, out var isInline))
            return obj.GetType().IsPrimitive || obj is IConvertible;

        if (isInline is true)
            return true;
        else if (isInline is false)
            return false;
        else
        {
            if (IsISingleMetaGraphNodeGen(obj, out var content))
                return IsInline(content);
            else if (IsIMultiMetaGraphNodeGen(obj, out var contents))
                return contents is null || contents.All(IsInline);
            else
                return false;
        }
    }

    private static bool TryGetIsInline(object obj, out bool? isInline)
    {
        if (obj.GetType().GetInterface("LINQPad.ObjectGraph.IMetaGraphNodeGen") is { } interfaceType)
        {
            isInline = (bool?)interfaceType.GetProperty("IsInline")!.GetValue(obj);
            return true;
        }
        else
        {
            isInline = null;
            return false;
        }
    }
}

LPRun shows the proper output:

word run
horizontal run
DumpContainer
Hyperlinq
Hyperlinq with action
Hyperlinq inside a DumpContainer
Hyperlinq with action inside a DumpContainer
OnDemand (which is Hyperlinq with action inside a DumpContainer)
OnDemand
{
  "DumpContainer": "DumpContainer inside something"
}
{
  "Hyperlinq": "Hyperlinq inside something"
}
This is the title: This is a separate line
title (outer): outer
{
  "A": "hello"
}
{
  "B": 5
}
{
  "C": "5"
}
{
  "D": "5"
}
{
  "E": "foo"
}
{
  "F": "world"
}
{
  "G": "bye"
}
{
  "H": "123 world hi"
}
{
  "H2": "123worldhi"
}
{
  "H3": "123 world hi"
}
{
  "H4": "123worldhi"
}
{
  "H5": "123 hi"
}
{
  "H6": "123 4 5 6 hi"
}
{
  "H7": [
    123,
    [
      4,
      5,
      6
    ],
    "hi"
  ]
}
{
  "I": "5 123 world hi"
}
{
  "I2": "hello 123 world hi"
}
{
  "J": "5 123 world hi"
}
{
  "K": [
    [
      "1",
      "5"
    ],
    [
      "2",
      123
    ],
    [
      "3",
      "world"
    ],
    [
      "4",
      "hi"
    ]
  ]
}
{
  "L": [
    "5",
    123,
    "world",
    "hi"
  ]
}
{
  "M": [
    "title (inner)",
    "inner"
  ]
}
{
  "N": [
    1,
    2,
    3
  ]
}
{
  "O": [
    1,
    [
      "title (inner)",
      "inner"
    ],
    3
  ]
}

I think that you can easily incorporate this code into TextOutputWriter.