Request: Add async `Util.Cache()` method

JeffMercado

The Util.Cache() method is synchronous, including the dataFetcher factory function.

Could an asynchronous version be added?

Current workaround is to cache an unpopulated serializable object (e.g., dictionary) and populate later.

JoeAlbahari

An async cache is normally implemented as a synchronous cache of tasks. For example:

string html = await Util.Cache (() => new HttpClient().GetStringAsync ("https://www.linqpad.net/robots.txt"));
html.Dump();

I'm not sure what an asynchronous version of Util.Cache would accomplish. Can you explain?

JeffMercado

Ah, didn't think to have it return Task<T>. That'll work.

JeffMercado

After using this more, I don't think caching the task is sufficient. If an exception occurs in the task, the failure is cached and it gets awkward trying to retry the call and replace the result. Had to restructure it completely.

JoeAlbahari

For specialized scenarios, you can create your own caching methods in My Extensions. If the dictionary is stored in a static field, it will persist between executions.

To illustrate a specialized scenario, here's the cache that LINQPad uses internally to cache NuGet package vulnerabilities. It periodically refreshes, can be saved to disk, and in the case of a fault, uses the previously cached value.

/// <summary>
/// Async refreshing cache that updates after a refresh only if the task doesn't fault.
/// </summary>
record FaultTolerantExpiringAsyncCache<TKey, TValue> (Func<TKey, TimeSpan?, CancellationToken, Task<TValue>> FetchFunc, TimeSpan CacheTime)
{
    Dictionary<TKey, CacheItem> _items = new ();
    DateTime _lastUpdate;

    public DateTime LastUpdate
    {
        get { lock (_items) return _lastUpdate; }
    }

    public void Deserialize (string json, JsonSerializerOptions options)
    {
        lock (_items)
            _items = new (JsonSerializer.Deserialize<KeyValuePair<TKey, CacheItem>[]> (json, options));
    }

    public string Serialize (JsonSerializerOptions options, TimeSpan maximumAge)
    {
        lock (_items)
            return JsonSerializer.Serialize (_items.Where (i => i.Value.Payload.IsCompletedSuccessfully && i.Value.LastUsed > DateTime.UtcNow - maximumAge).ToArray (), options);
    }

    public bool GetIfComplete (TKey key, bool used, out TValue value)
    {
        lock (_items)
        {
            if (_items.TryGetValue (key, out var item) && item.Payload.IsCompletedSuccessfully)
            {
                if (used) item.LastUsed = DateTime.UtcNow;
                value = item.Payload.Result;
                return true;
            }
            else
            {
                value = default;
                return false;
            }
        }
    }

    public Task<TValue> Get (TKey key)
    {
        lock (_items)
        {
            if (_items.TryGetValue (key, out var item))
            {
                if (DateTime.UtcNow > item.Expires)
                {
                    // Refresh the cache for the benefit of future calls (but don't wait for the refreshed item).
                    var cts = new CancellationTokenSource ();
                    item.Refresh (FetchFunc (key, DateTime.UtcNow - item.Expires, cts.Token), cts, CacheTime);
                    _lastUpdate = DateTime.UtcNow;
                }
                item.LastUsed = DateTime.UtcNow;
            }
            else
            {
                var cts = new CancellationTokenSource ();
                _items.Add (key, item = new CacheItem (FetchFunc (key, null, cts.Token), cts, CacheTime));
                _lastUpdate = DateTime.UtcNow;
            }
            return item.Payload;
        }
    }

    /// <summary>Pokes an already-available value into the cache.</summary>
    public void Update (TKey key, TValue value, bool persist)
    {
        lock (_items)
        {
            if (_items.TryGetValue (key, out var item))
            {
                item.SetResult (value, CacheTime);
                if (persist) item.LastUsed = DateTime.UtcNow;
            }
            else
                _items.Add (key, item = new CacheItem (value, CacheTime) { LastUsed = persist ? DateTime.UtcNow : DateTime.MinValue });

            _lastUpdate = DateTime.UtcNow;
        }
    }

    class CacheItem
    {
        readonly object _lock = new ();
        TaskCompletionSource<TValue> _valueSource = new ();
        CancellationTokenSource _cts = new ();

        public DateTime Expires { get; set; }  // set is public for JSON serializer

        public DateTime LastUsed { get; set; } = DateTime.UtcNow;

        [JsonIgnore]
        public Task<TValue> Payload => _valueSource.Task;

        /// <summary>For JSON serialization</summary>
        public TValue JsonContent
        {
            get => _valueSource.Task.Result;
            set => SetResult (value, null);
        }

        /// <summary>For JSON serialization</summary>
        public CacheItem ()
        {
        }

        public CacheItem (TValue value, TimeSpan cacheTime)
        {
            _valueSource.SetResult (value);
            Expires = DateTime.UtcNow + cacheTime;
        }

        public CacheItem (Task<TValue> valueTask, CancellationTokenSource cts, TimeSpan cacheTime) =>
            Refresh (valueTask, cts, cacheTime);

        public void SetResult (TValue value, TimeSpan? cacheTime)
        {
            lock (_lock)
            {
                if (cacheTime.HasValue) Expires = DateTime.UtcNow + cacheTime.Value;
                _cts.Cancel ();
                if (!_valueSource.TrySetResult (value))
                {
                    var newSource = new TaskCompletionSource<TValue> ();
                    newSource.SetResult (value);
                    _valueSource = newSource;
                }
            }
        }

        public void Refresh (Task<TValue> valueTask, CancellationTokenSource cts, TimeSpan cacheTime)
        {
            lock (_lock)
            {
                Expires = DateTime.MaxValue;
                _cts.Cancel ();
                _cts = cts;
                valueTask.ContinueWith (_ =>
                {
                    lock (_lock)
                    {
                        if (cts.IsCancellationRequested) return;

                        if (valueTask.IsCompletedSuccessfully)
                            SetResult (valueTask.Result, cacheTime);
                        else
                        {
                            if (Expires == DateTime.MaxValue) Expires = DateTime.UtcNow;
                            // TrySetException will fault only if there's no existing cached value.
                            if (valueTask.IsFaulted) _valueSource.TrySetException (valueTask.Exception.InnerException);
                            // Ignore task cancellation.
                        }
                    }
                });
            }
        }
    }
}

doggy

The LINQPad 8.1 already have a Util.CacheAsync method now.

JoeAlbahari

That's right - new Util.CacheAsync method ensures that faulted tasks are not cached.