BlockingCollection.GetConsumingEnumerable and Parallel.ForEach hang

BlockingCollection.GetConsumingEnumerable and Parallel.ForEach hang by Mike Barry

Closed

as Won't Fix Help for as Won't Fix

Type:	Bug
ID:	674705
Opened:	6/8/2011 2:24:16 PM
Access Restriction:	Public
Moderator Decision:	Sent to Engineering Team for consideration

Workaround(s)

view

User(s) can reproduce this bug

Using a blocking collection GetConsumingEnumerable with Parallel.ForEach could never finish if only 1 item at a time is ever in the collection. (This is probably not the only case, just the first one I found.)

Details (expand)

Visual Studio 2010 SP1

Run the following code on a dual core machine and it should hang before it prints Loop Done. On my machine it hangs between 4 and 6 every time.

BlockingCollection<int> blockingCollection = new BlockingCollection<int>();
            int? lastInserted = null;

            Task readBlockingTask = new Task(() =>
                                                 {
                                                     Parallel.ForEach(blockingCollection.GetConsumingEnumerable(), x =>
                                                                                                                     {
                                                                                                                         Debug.Assert(lastInserted.HasValue && lastInserted.Value == x);
                                                                                                                         Console.WriteLine("Consuming {0}", x);
                                                                                                                         lastInserted = null;
                                                                                                                     });
                                                 });

            readBlockingTask.Start();

            for (int i = 0; i < 1000; i++ )
            {
                lastInserted = i;
                Console.WriteLine("Adding {0}", i);
                blockingCollection.Add(i);

                while (lastInserted.HasValue)
                {
                    Thread.Sleep(100);
                }
            }

            Console.WriteLine("Loop done");

English

Windows XP

English

Loop doesn't finish

Loop finishes

0 attachments

Posted by Daniel Crenna on 11/19/2012 at 9:39 PM

The provided workaround is not acceptable as it requires all derivative works to comply with the MS-LPL, in other words, be completely tied down to Windows environments. This is not an OSI-approved license and accepting the workaround as given introduces all kinds of headaches for any solutions that ship cross-platform. The entire Rx framework is licenses under and OSI-approved license, as is ASP.NET. Please update the TPL examples to work with a real OSS license so that we can actually take your advice without introducing risk and vendor constraints our customers don't want.

Posted by Mike Barry on 8/22/2012 at 1:45 PM

Was this fixed in .NET 4.5?

Posted by Mike Barry on 6/9/2011 at 1:32 PM

Thanks. You guys rock as usual!

Posted by Microsoft on 6/9/2011 at 12:46 PM

Mike,

This is a known problem. The issue is that the partitioning mechanism used in Parallel.ForEach consumes elements in "chunks"; chunk size starts at 1, then gradually grows. The internal logic grabs all elements in a chunk and then hands them back to Parallel.ForEach to process them as a group. In your example, for any chunk size greater than 1, the enumerator will hang because it can't grab the second element in the chunk until it processes the first element in the chunk.

See http://blogs.msdn.com/b/pfxteam/archive/2010/04/06/9990420.aspx for a discussion of this problem, as well as some code (GetConsumingPartitioner() extension method and BlockingCollectionPartitioner<T> class) that allow you to work around the problem.

(And yes, we will add a workaround out-of-the-box in an upcoming version of .NET).

Hope this helps,
--Joe

Posted by MS-Moderator09 [Feedback Moderator] on 6/8/2011 at 7:41 PM

Thank you for submitting feedback on Visual Studio 2010 and .NET Framework. Your issue has been routed to the appropriate VS development team for review. We will contact you if we require any additional information.

Posted by MS-Moderator01 on 6/8/2011 at 2:54 PM

Thank you for your feedback, we are currently reviewing the issue you have submitted. If this issue is urgent, please contact support directly(http://support.microsoft.com)