Home Dashboard Directory Help
Search

Directory.EnumerateDirectory etc unusable due to frequent UnauthorizedAccessExceptions (even runas Administrator) by Schneider


Status: 

Active


46
0
Sign in
to vote
Type: Bug
ID: 512171
Opened: 11/19/2009 5:03:53 AM
Access Restriction: Public
6
Workaround(s)
view
27
User(s) can reproduce this bug

Description

When running as non-administrator, the new set of File system Enumeration APIs cannot be used to recursively enumerate files or folders in a reliable way.

These are all affected:
Directory.EnumerateDirectories
DirectoryInfo.EnumerateDirectories
Directory.EnumerateFiles
DirectoryInfo.EnumerateFiles
Directory.EnumerateFileSystemEntries
DirectoryInfo.EnumerateFileSystemInfos

By design it appears these file system enumerators throw exceptions when they encounter an inaccessible folder, instead of just ignoring it and moving on. This exception causes the enumeration to cease.

This may be due to a limitation of the Win32 API, but something here seems wrong. For example, a non-administrator can enumerate the file system by doing dir /s at the command line, so we should surely be able to do the same from .NET code?
Details
Sign in to post a comment.
Posted by nan02 on 10/18/2013 at 4:15 AM
Still an issue in .Net 4.5
Posted by Xied75 on 11/30/2012 at 8:21 AM
Still get this on Windows Azure Worker Role on Windows Server 2012.
Posted by PCTO on 9/6/2011 at 3:35 PM
Yes, the best solution is an option that allows the developer to choose if he or she wishes the enumeration to continue if an authorization exception is encountered.

As is the methods are completely unprofessional and unusable in a production application, and should either be adjusted, or deprecated and replaced. The primary concern is that they essentially deny access to discovering files that the current user DOES have permission to access by short-circuiting just because a single file or folder exists to which that user does not have access. The user should have an easy means for enumerating all of the files in the file system to which he/she has permissions, regardless of any files that may exist to which they do not.

This is possible with the windows API, and currently the only viable solution to this issue is to write your own recursion using pinvoke calls, which is not at all ideal. It is my opinion that this is the single most poorly implemented piece of functionality in the entire framework.
Posted by Richard Deeming on 8/22/2011 at 9:35 AM
This problem also applies to junction points.

For example, attempt to enumerate the directories in your profile folder. You will get:
Access to the path '%USERPROFILE%\AppData\Local\Application Data\' is denied.

In this case, you don't want to enumerate the junction point, since it points back to the parent directory, so you'd end up with infinite recursion.
Posted by Ramon Barreda on 1/6/2011 at 11:18 AM
Please include a fix with Visual Studio 2010 SP1 as these are very powerful functions, and not yet fixed in Visual Studio SP1 Beta.

The workarounds suggested are not acceptable, as both take 5 to 6 minutes to iterate just through my users folder with > 25,000 files. (Even using Parallel programming)

Right now there are two choices for me:

Either write very complex methods for “IEnumerable-based methods” allowing to catch SecurityExceptons, and then process those “Access Denied” Folders using Shell API calls.
(almost done)

Or use only Shell API calls (FindFirstFile & FindNextFile). Results return in about 1 minute.
(currently used by my backup and file organizer applications)

Justin Van Patten suggested a “new SearchOption - continue on errors”, but there would be no way to know which files or folders were denied access. So full iteration of files and folders would not be possible.

I think possible solutions could be: Callback to catch the exception(s) thrown by the “IEnumerable-based methods” without breaking the loop, or perhaps a Security parameter that allowed the iteration of the whole tree. (Overloaded methods with extra parameters).

Thank You.
Posted by Daniel J. Wojcik on 12/23/2010 at 11:12 AM
The workaround pointed by Frank Dzaebel works.
What he didn't mention is that it is 15 times as slow. (2000 files/second vs 120 files/second on my machine)

But, not much choice in the matter, is there.
Posted by Schneider on 4/13/2010 at 3:14 AM
Thanks for response.

Shame it didn't make it into .net 4 but I understand the complexity of planning this stuff.

That said it seems a bit of an obvious oversight not to have this as a basic user story - somebody trying to write a basic 'explorer' style application that asynchronously enumerates through the file system.

Your comment about SearchOptions is a very interesting one from a design perspective. Regarding principle of least surprise I think most ppl would be surprised that you could not enumerate any directory containing any trivial permission violation. Hence having to explicitly pass a parameter is odd. Changing it now of course could potentially be a breaking change? Lastly, semantically I'm not sure "errors" is the right description for this - I dont consider it an error if I cant enumerate a file due to permissions. As I said using dir *.* /s does not throw "errors". Anyway, just food for thought.

Hope to see this fixed in SP1.

Thanks, Schneider
Posted by Microsoft on 3/31/2010 at 5:12 PM
Hi Schneider1,

Thanks for reporting this. One of the goals of the new IEnumerable-based methods on Directory (EnumerateFiles, EnumerateDirectories, and EnumerateFileSystemEntries) was to have consistent behavior with the older array-based methods (GetFiles, GetDirectories, and GetFileSystemEntries), to make it easy to move from the old methods to the new ones. The behavior you're seeing is consistent with the older methods.

That said, we agree that it'd be useful to be able to more easily enumerate all files/directories on the file system that an app can access. Unfortunately, we weren't able to address this for .NET 4, but we are considering addressing this in the next version of .NET. One way we might address this is by providing a new SearchOption value to "continue on errors". Therefore, I'm going to keep this suggestion open for tracking.

Regards,

Justin Van Patten
Program Manager
CLR Base Class Libraries
Posted by Microsoft on 11/20/2009 at 8:05 PM
Thank you for your feedback, We are currently reviewing the issue you have submitted. If this issue is urgent, please contact support directly(http://support.microsoft.com)
Posted by Schneider on 11/19/2009 at 5:15 AM
In fact, even when running as an administrator the example code stops because the entire file system is enumerated. On my computer it stops on this:

UnAuthDir: Access to the path 'c:\Documents and Settings\' is denied.
Sign in to post a workaround.
Posted by Sh0gun on 12/27/2013 at 9:20 AM
For the meantime, AlphaFS can probably help you out:

https://alphafs.codeplex.com/discussions/478008
Posted by nan02 on 10/19/2013 at 8:34 AM
I found this solution at Stackoverflow a perfect replacement:
http://stackoverflow.com/a/9758932/368354

Can be used exactly the same as the Directory.* methods and works as expected.
Posted by Sandro Engelmann on 10/7/2011 at 11:25 AM
Hi "Schneider",

I had the same problem and re-implemented the functionality. You can find the solution at http://rwtools.codeplex.com/.

Inside are classes like "DirectoryEnumerator" and "FileEnumerator" which ignores errors, or (if sombody likes) throws the error and continue iterating.

Hope it helps.

Regards,
Sandro
Posted by Lionel KOMSA on 3/27/2011 at 8:45 AM
I done a misstake in my code because I take care of searchPattern to explore subDirectories. Consequently, if a subdirectory doesn't match with searchPattern, I will not explore it.

I wrote new version to correct this, but I must doing two request on each directory : one to get items whose match searchPattern and another to get all subdirectories... Therefore, performance are a little worse than before. So, I add a version without any searchPattern, to get all items, and in this version, only one request on each directory, so good performance !!

Those new version are non recursive and conserve traditional order.
For information : my method without searchPattern are more faster than Framework.Net one !!! (but don't really know why ;o)

You can get my test solution here : http://lionel.komsa.free.fr/temp/TestDirEnumerate2.zip
Or look here for more details : http://social.msdn.microsoft.com/Forums/en-US/netfxbcl/thread/d1988cec-b74c-4375-9b1b-6929a8a724dd/

Lionel
Posted by Lionel KOMSA on 3/11/2011 at 4:27 PM
Hi

Forgive my poor english.
I wrote functions to work around this problem; they take a delegate function Func<DirectoryInfo, Exception, bool> in parameter to callback when Enumerate a directory throw an exception. If this delegate return true, the exception is ignored.

For more details and source code, look at my post here : http://stackoverflow.com/questions/1118568/how-do-i-get-a-directory-size-files-in-the-directory-in-c/5279770#5279770

You can too download my test solution here : http://lionel.komsa.free.fr/temp/TestDirEnumerate.zip
The function are in IO.DirectoryInfoExtension. There's "EnumerateFiles_Recursive", "EnumerateFiles_NonRecursive" (use a queue to avoid recursion) and "EnumerateFiles_NonRecursive_TraditionalOrder" (use a stack of queue to avoid recursion and keep traditional order : content of directory and then content of first subdirectory and his own subdirectories and then content of the second...). Keep those 3 methods has no interest, I write them only for test the best one. I think to keep only the last one. I also wrote the equivalent for EnumerateFileSystemInfos and EnumerateDirectories.

Hope this help.
Sincerely.

Lionel KOMSA
Posted by Frank Dzaebel on 11/17/2010 at 7:15 AM
Try to use the following documentation proposals:

[How to: Iterate Through a Directory Tree (C# Programming Guide)]
http://msdn.microsoft.com/en-us/library/bb513869.aspx