Tasks returned by 'async' methods should be serializable (Async CTP) - by Stefan Wenig

Status : 

  By Design<br /><br />
		The product team believes this item works according to its intended design.<br /><br />
		A more detailed explanation for the resolution of this particular item may have been provided in the comments section.


8
1
Sign in
to vote
ID 620941 Comments
Status Closed Workarounds
Type Suggestion Repros 0
Opened 11/11/2010 3:28:08 AM
Access Restriction Public

Description

The types Task and Task<> should be serializable, including continuation delegates created by 'async' methods. 

This would enable us to use continuation passing style programming for long-running methods, such as workflow engines, or for ASP.NET apps using state servers!

The compiler should mark the delegate objects holding the captured variables as serializable if all captured variables are serializable.
Therefore, the types AsyncMethodBuilder and TaskAwaiter would have to be serializable too.

Alternatives: 
- There could be a syntax to tell the compiler to make delegates serializable, such as an attribute on the async method. 
- The compiler could choose to always mark them as serializable. Rationale: These types are only visible at run-time anyway, and serialization might actually succeed due to a serialization surrogate only registered at run-time.

This is similar to https://connect.microsoft.com/VisualStudio/feedback/details/278510/support-serialization-of-iterator-state
Sign in to post a comment.
Posted by Stephen [MSFT] on 12/23/2010 at 1:43 PM
Hi Stefan-

Thanks for the additional info. Given that you agree that there are problems with making tasks serializable, I suggest you open a new connect suggestion to ask for the core thing you want, which is to be able to substitute your own builder and return type. There are of course some difficulties in making that happen (and in making it happen to the extent that you want, including making all of the state machine scaffolding generated by the compiler itself serializable), but it'd be better discussed as its own connect bug.

Happy holidays,
Stephen
Posted by Stefan Wenig on 12/23/2010 at 7:34 AM
Oh, and thanks for giving this so much thought!

Stefan
Posted by Stefan Wenig on 12/23/2010 at 7:34 AM
Hi Stephen,

let me try to be a bit more specific.
1. I don't try to do async programming here. I'm just using the general purpose nature of the new async/await features.
2. Continuation-passing style programming is not a very specialized thing. Eric Lippert just went out of his way on his blog go explain how it can be used to do a variety of things. And the way you did it is very much on the generalized side of things, except for the naming and of course the marketing. Just like Eric and friends came forward after finishing LINQ, explaining how it is just a monad and all, and how it can be used to do all sorts of things, I expect the same for async/await. You chose names that make it comprehensible for the most likely scenario, which is a good thing. Now I just hope that doesn't make you think it should be limited to async.
3. CPS-style web programming is actually quite popular outside the .net/java mainstream: Paul Graham wrote a seminal paper, and here's a start: http://en.wikipedia.org/wiki/Continuation#Continuations_in_Web_development. it's definately popular with the lambda-the-ultimate crowd!

It's a pity that async is hard-coded to the Task<T> and AsyncMethodBuilder etc. classes. And I do see the problems that could arise from making Task and AsyncMethodBuilder serializable. I'm considering posting an alternative feature request asking that instead of requiring a return type of void or Task<T>, an async method should just require any return type that brings an equivalent for AsyncMethodBuilder with it. Like that:

[Serializable]
[AsyncBuilder (typeof (MyCPSMethodBuilder<>))]
class MyCPSTask<T>
{
}

these types would then be expanded by the translation and would just need to support whatever ctors and methods the current types support: static Create(), SetResult(T), SetException(ex) etc. (Just like foreach will take any object that has GetEnumerator(), or LINQ query expressions will take any objects that have Where, Select etc. methods defined for them.

That'd be cleaner than just making types serializable in limited scenarios, but also more sophisticated I guess. But the idea grows on me. Why artificially limit such a generic feature to such a narrow usage just by hard-coding class names?

Finally, here's where I would serialize: For web apps, I'd serialize at the end of the request, and deserialize at the beginning of the next one. In my simple loop sample, that'd be:

     var formatter = new BinaryFormatter();
     var stream = new MemoryStream();

     var runner = new Runner();
     Task<int> task = runner.Foo(3);
     formatter.Serialize (stream, runner);
    
     bool cont = true;
     while (cont)
     {
        runner = (Runner) formatter.Deserialize (stream);
        cont = runner.Continue();
        formatter.Serialize (stream, runner);
        runner = null;
     }


And yes, you're right, the closure class would need to be serializable too. Maybe as an explicit opt-in would be cleaner, but I could live with the [Serializable] attribute being applied whenever all captured variables are serializable. Capturing a non-serializable local variable would naturally break serialization of the whole thing, that would not come too unexpected.

I realize you consider this an esoteric feature, but I think it really is not. And now would be the time to design C#5 in a way that just stays in line with it's current philosophy of carefully introducing general-purpose language elements that just "accidentially" help solving current problems (like the impedence mismatch or async coding).
Posted by Stephen [MSFT] on 12/21/2010 at 4:40 PM
Hi Stefan-

It sounds like you have a fairly specialized scenario. Even if it would happen to work in your case, making a task serializable would lead to a host of problems for the most common cases. The only way it would work "correctly" is if the task is serialized after it completes... if it's serialized before it completes, which would be a common case (and there'd be a race most folks would not be aware of), then the deserialized copy would never complete, even if the original task did. It's not clear to me conceptually how this would actually be useful to you... if your task has already completed when you want to serialize it, then you can just serialize the Result, and if you really want a task object upon deserialization, create a new task object that wraps the Result; if your task has not already completed when you want to serialize it, then you'll be in the same negative situation I previously outlined, where the original task completing will have no bearing on the deserialized task, which will never complete. (As an aside, note, too, that the display class generated by the C# compiler for closures is not currently serializable.)

That said, I did take a quick look at your example. When would you expect to do the serialization? Your example seems to fall into the same problem I outline above. At the beginning of your Main, you call Foo to retrieve a Task<int>. And at the end of the Main function, you access that Task<int>'s Result. If before that task completes (which won't happen until the final "return level;" statement of your async method, which won't happen until you call runner.Continue() the appropriate number of times) you do the serialization, the deserialized task will never complete, because it'd be a totally separate Task<int> from the one originally returned from Foo() that will complete when you hit that "return level;" statement in Foo.

I hope that helps. And regardless, thanks for the feedback.
Posted by Stefan Wenig on 12/20/2010 at 7:53 AM
I tried to attach a file, but using two different browsers, it didn't work. It's not too long, so I'll just post it here:

using System;
using System.Threading.Tasks;
using System.Diagnostics;
using System.Text;

namespace Await
{
[Serializable]
public class Runner
{
    public async Task<int> Foo(int level)
    {
     if (level == 0)
        return 0;

     var indent = new StringBuilder().Append(' ', 4 - level).ToString();
     Console.WriteLine(indent + "starting...");

     await Exec (indent + "1");
     await Exec (indent + "2");
     var x = await Foo(level - 1);
     await Exec (indent + "3");

     return level;
    }

    public Runner Exec (string s)
    {
     Console.WriteLine (s);

     return this;
    }

    [DebuggerHidden]
    public bool Continue()
    {
     if (_continuation == null)
        return false;

     _continuation();
     return true;
    }

    private Action _continuation;

    [DebuggerHidden]
    public Runner GetAwaiter()
    {
     return this;
    }

    [DebuggerHidden]
    public bool BeginAwait(Action continuation)
    {
     if (_continuation != null)
        throw new InvalidOperationException("BeginAwait without EndAwait");

     _continuation = continuation;
     return true;
    }

    [DebuggerHidden]
    public void EndAwait()
    {
     _continuation = null;
    }
}


    class Program
    {
        static void Main(string[] args)
        {
     var runner = new Runner();
     Task<int> task = runner.Foo(3);

     for (; ; )
     {
        Console.WriteLine("     Continue()");
        if (!runner.Continue()) // runner2
         break;
     }

     Console.WriteLine("result: {0}", task.Result);
        }
    }

}
Posted by Stefan Wenig on 12/20/2010 at 7:45 AM
Hi Stephen!

Thanks for your detailed reply. I disagree though. Making tasks serializable would definitely do the trick in my scenario.

Yes, it would create a new task, a different object. That's how serialization works. But in my scenario, the task would not actually be executed asynchronously, it just uses async/await for continuation-passing style programming.

The deserialized task would then just take the closure (the captured variables), pass it to the continuation delegate, and work on.

Yes, we could do something similar via WF to achieve something similar. But in the end, anything I could want out of C# could be achieved using some other tool that supports that, right? CPS-style Web or WF-programming in C# would be a very cool trick, showing off some of the real power of C#. And yes, we actually have production code that does that now, only with _very_ verbose syntax, and we would actually benefit a lot from that feature.

I'll attach a small C# sample, badly designed and oversimplified, that shows what we'd like to achive. It just executes a function step by step in a loop. Could be a message loop, web requests, WF activities or anything else though. Only for web requests, it would not support state servers for lack of serialization. (useless for WF too.)

Cheers,
Stefan
Posted by Stephen [MSFT] on 12/10/2010 at 9:34 AM
Sincerely,
Stephen Toub
Microsoft
Posted by Stephen [MSFT] on 12/10/2010 at 9:33 AM
Hi Stefan-

Thanks for taking the time to provide feedback! Much appreciated.

Making Tasks serializable unfortunately wouldn't help in the cases you've outlined. When you invoke an async method, it starts running immediately, before the task is even returned. At least all of the code up until the first await is executed synchronously as part of the method call. As such, the task that's returned represents an already running operation. If it were serialized and then unserialized, the resulting unserialized task would be a clone of the original task that would be a completely different object. When the async operation already underway actually completed, it would complete the original task, and the cloned task would be none-the-wiser, never completing.

If you want the kind of semantics you're describing, it would be better to utilize a system like Windows Workflow Foundation, which has these kinds of persistence semantics built in (note, too, that there are times during an activities lifecycle, in particular when async operations like this are outstanding, that the workflow can't be persisted). You can utilize async methods within such workflows, just as you would utilize existing Asynchronous Programming Model (Begin/End) implementations or existing Event-based Async Pattern (e.g. WebClient) implementations.

Thanks, again, for the suggestion.
Posted by Microsoft on 11/11/2010 at 4:20 AM
Thank you for your feedback, we are currently reviewing the issue you have submitted. If this issue is urgent, please contact support directly(http://support.microsoft.com)