Executing commands which require quotes and variables is practically impossible - by Joel Bennett

Status : 

  Fixed<br /><br />
		This item has been fixed in the current or upcoming version of this product.<br /><br />
		A more detailed explanation for the resolution of this particular item may have been provided in the comments section.


116
1
Sign in
to vote
ID 376207 Comments
Status Closed Workarounds
Type Bug Repros 25
Opened 10/17/2008 12:10:35 PM
Access Restriction Public

Description

It's FAR TOO HARD to pass parameters to applications which require quoted strings.  I asked this question in IRC with a "roomful" of PowerShell experts, and it took hour for someone to figure out a way (I originally started to post here that it is simply not possible).

This completely breaks PowerShell's ability to serve as a general purpose shell, because we can't do simple things like executing sqlcmd. The number one job of a command shell should be running command-line applications...

As an example, trying to use SqlCmd from SQL Server 2008, there is a -v parameter which takes a series of name:value parameters.  If the value has spaces in it, you must quote it. For instance, this will work in cmd.exe (it doesn't do anything, just exits silently):
        sqlcmd -v user="Joel Bennett" -Q "select '$(user)' as UserName"

But in PowerShell, it has to be:
        sqlcmd -v user=`"Joel Bennett`" -Q 'select ''$(user)'' as UserName'

Notice that you have to escape things two different ways here?  How many people are going to figure that out?

But it gets worse if you need to user a variable (eg, $Env:FullName):
$Env:FullName="Joel Bennett"

So you can execute it in CMD: 
      sqlcmd -v user="%FullName%" -Q "select '$(user)' as UserName"

But how many PowerShell users are going to figure out that this (sortof) works:
      sqlcmd -v user=($env:FullName) -Q 'select ''$(user)'' as UserName'

What appears to be happening is that "user=($env:FullName)" is treated as a two part array: "user=" and "Joel Bennett" ... so the engine inserts a space after the "user=" (which I can't get rid of by setting $ofs, oddly) and then inexplicably and magically chooses to wrap quotes around the "Joel Bennett" ....

The problem is, as you can see from these examples, there is no single way to write a command line to invoke this application correctly, so even after you master all 4 or 5 different ways of quoting and escaping things, you're still guessing as to which will work when ... or, you can just shell out to cmd, and be done with it.
Sign in to post a comment.
Posted by Tonči Grgin on 3/20/2015 at 12:09 PM
Porting "EVAL" from bash. For testing, want to copy archive, unpack *.tar.gz and remove both files (*.tar and *.tar.gz) using 7z. 7z is installed in regular place. C:\Program Files\7-Zip\7z.exe x some.tar.gz | Out-Null does NOT work no matter what one does (escaping, quoting...).

Quickly put up some script code:
1) Find 7z executable:
    $TAR = [regex]::Escape((Get-Command 7z.exe).Path) - No-Go
    $TAR = "'" + (Get-Command 7z.exe).Path + "' x" ... - No-Go
    Only putting 7z in PATH works ($TAR = '7z.exe').
2) Compile the command line:
    unpack_tarball 'blahblah-0.4.12.7.tar.gz'
...
    #Extract
    exec_command "$TAR" 'x' "$TARBALL_NAME" '| Out-Null'

3) Mimic EVAL with exec_command function:
function exec_command {
    <#
    .SYNOPSIS

    Method to execute any command and verify command was a success.
    #>
    
    $ToExec=""
    for ( $i = 0; $i -lt $args.Length; $i++)
    {
            $ToExec = ($ToExec,$args[$i]) -join ' '
    }

    $sb = [scriptblock]::Create($ToExec)
    Invoke-Command -ScriptBlock $sb -ErrorVariable errmsg 2>$null
    
    if ( $errmsg )
    {
        exit 1
    }
}

I can work around this but it's PAIN and never a good thing to change approach on case by case basis...
Posted by piers7 on 1/13/2015 at 5:31 PM
Look I know this is incredibly frustrating. I've burnt way too many hours with the SQLCMD parameters pain. But sadly the outcome is very simple:

- only exe's that follow the Shell32::CommandLineToArgvW Windows quoting/escaping conventions can be used directly by PowerShell, or via Start-Process
- anything that deviates from these will need a single *literal* argument string constructed and passed to the exe using [Process]::Start. Constructing this string within PowerShell may require some double-escaping, depending on context.
- For a fixed number of arguments you can also use --% and reference environment variables, as Keith demonstrates below.

Anything that presents an API where arguments are passed as an array (PowerShell, or Start-Process) is *always* going to run into edge cases, since Windows processes are not actually passed an array of arguments (they're passed a single string, and *normally* interpreted by Shell32::CommandLineToArgvW)

It is amazingly frustrating that SQLCMD and MSDEPLOY are the worst culprits here, given the high frequency with which they are automated using PowerShell. Perhaps a better campaign would be for these applications to fix their command-line parsing conventions...
Posted by michael12345 on 7/18/2014 at 12:47 PM
I too have the challenge of trying to write a reasonably complicated MSDeploy.exe commandline. I'm testing it in PowerShell, but I don't have 4 hours to figure out how to do the correct escaping so I'll be starting up cmd.exe and ignoring PS for this task.
Posted by Douglas McCormick on 2/8/2014 at 1:42 PM
I have to agree with all the other comments/posts below.........This is complete garbage Microsoft, really? Powershell is that dumb and you consider this as "working as designed"?
How in the world would this bug be considered closed in any definition-- I'm ready to take a hammer to PS--- I'm looking to wrap nearly everything into cygwin instead-- at least there's enough people and thought leadership to fix a bug/issue instead of conveniently calling it closed.

This should be addressed properly and patched up for v2 and above

Posted by Keith Hill MVP on 2/3/2014 at 9:18 PM
>> --% operator presented as a fix to this issue is totally inadequate for any but the most trivial of use cases.

Huh? This is for way more than trivial use cases. It essentially puts the PowerShell parsing into a cmd.exe equivalent mode (ie dumb mode) e.g.:

MSBuild /t:Publish --% /p:TargetDatabaseName="MyDatabase";TargetConnectionString="Data Source=.\;Integrated Security=True" /p:SqlPublishProfilePath="Deploy.publish.xml" Database.sqlproj

Or how about this case:

Set-Content EchoTest.cmd "Echo %1"
$test="Hello There"
$env:test = $test
.\echotest.cmd --% %test%

Outputs:
Hello

Or how about this case:

$env:SiteName = $SiteName
$env:SiteManagementUrl = $SiteManagementUrl
MSDeploy.exe --% -verb:delete -verbose -dest:contentPath="%SiteName%/App_Offline.htm",computerName="%SiteManagementUrl%"

If you understand how --% works, it isn't very difficult to get your exe parameters to work just as they did in cmd.exe.
Posted by stangm on 11/15/2013 at 7:46 AM
Just to add, this is not fixed in powershell 3. The automatic entry of quotes when spaces are in the variable is major problem. Example:

Set-Content EchoTest.cmd "Echo %1"
$test="Hello There"
.\echotest.cmd $test

Expected Output:
Hello

Actual Output:
"Hello There"

Note the quotes

Now try
Set-Content EchoTest.cmd "Echo %1"
$test="Hello"
.\echotest.cmd $test

Expected Output
Hello

Actual Output:
Hello

Note no quotes

The moment you put a space in a string, Powershell "helpfully" adds quotes to it when passing to the command.... Why?




Posted by Morgan Larosa on 9/3/2013 at 11:14 PM
Just adding my voice to the chorus of "this isn't actually fixed" - the --% operator presented as a fix to this issue is totally inadequate for any but the most trivial of use cases.

The most common complaint that I've seen in relation to this issue is Web Deploy: if I have, for example, a runCommand provider issuing a command with parameters, there's no way for me to successfully run the msdeploy command line from Powershell if I want to generate any of the arguments using variable interpolation or string formatting.

The evidence is all here in the previous comments and the thousands of forum threads, blog posts, StackOverflow questions, etc. - this issue is not fixed, and to mark it as such is disingenuous and disappointing from the Powershell team.
Posted by Mr Konrad Sikorski on 7/31/2013 at 5:14 AM
I cannot find out the correct way to invoke something like this:

MSBuild /t:Publish /p:TargetDatabaseName="MyDatabase";TargetConnectionString="Data Source=.\;Integrated Security=True" /p:SqlPublishProfilePath="Deploy.publish.xml" "Database.sqlproj"

I am using this command:

$DatabaseConnectionString = "Data Source=.\;Integrated Security=True"
MSBuild /t:Publish "/p:TargetDatabaseName=`"$DatabaseName`";TargetConnectionString=`"$DatabaseConnectionString`"" /p:SqlPublishProfilePath=`"Deploy.publish.xml`" `"Database.sqlproj`"

and it doesn't work. I have spend 4 hours trying many solutions without success. :/
Posted by Mike Cheel on 7/18/2013 at 11:03 AM
Until you can call a powershell script with spaces in the parameter variables this is not fixed at all. It is a shame I have to call a batch file from powershell just to call msdeploy.exe.
Posted by ColinBowern on 6/24/2013 at 10:10 PM
Here is an example of something that I can't do in PowerShell 3.0:

MSDeploy.exe -verb:delete -verbose -dest:contentPath='$SiteName/App_Offline.htm',computerName='$SiteManagementUrl'

This attempt fails:

& $MSDeploy -verb:delete -verbose $("-dest:contentPath='{0}',computerName='{1}'" -f $ContentPath, $SiteManagementEndpoint)

with:

MSDeploy.exe : Error: Unrecognized argument '"-dest:contentPath='My Site/App_Offline.htm',computerName='https://webnode:8172/MSDeploy.axd?Site=My+Site'"'. All arguments must begin with "-".
Posted by emission on 12/10/2012 at 2:38 AM
Closed?

Can MS please describe:

1. What you mean by closed.
2. What's so difficult... This seems like shell language 101.
Posted by matthickford on 6/6/2012 at 8:07 AM
"Closed as fix"?

If so, PLEASE link the fix!!!!!!
Posted by AussieJohn on 6/1/2012 at 12:12 AM
If we could call PowerShell directly from SQL Server (SQL Connect 303417) that would help a fair bit with SQL PowerShell quoting issues when going in that direction. To have to call xp_cmdshell (so the old CMD shell) and then PowerShell adds to quoting problems.

In the PowerShell to SQL direction, an easy-to-fix sqlcmd example is the server name. In CMD shell
sqlcmd -S machine\instance -E
works. In PowerShell you have to do
sqlcmd -S 'machine\instance' -E
which may inhibit some DBAs from using PowerShell.
Posted by Microsoft on 12/9/2011 at 4:24 PM
Using the LiteralPath parameter is the pattern we've established for dealing with special characters. We've added LiteralPath to a bunch of cmdlets in PowerShell 3.0. We've also addressed a number of one-off cases in our backtick and square bracket handling. We realize we've not done here, so please continue to file additional specific high-impact cases so we can continue to address them in future releases.
Posted by enndub on 7/15/2011 at 5:04 PM
You can escape double quotes for command line arguments with a backslash.

PS C:\Documents and Settings\Nick> echo '"hello"'
"hello"
PS C:\Documents and Settings\Nick> echo.exe '"hello"'
hello
PS C:\Documents and Settings\Nick> echo.exe '\"hello\"'
"hello"

Much easier than most of the workarounds here, but it would still be nice to get this bug fixed.

As long as we pass powershell a properly escaped string, it should automatically take care of whatever escaping is necessary for however it executes command line programs.
Posted by Softlion on 12/14/2010 at 10:22 PM
The first bug is:

$toto = "toto"
$test = ($toto,'-value='+$toto)
$test

Result by Powershell v2:
toto
-value=
toto

The correct result should be:

toto
-value=toto


The second bug is:

# Needs poweshell community extentions installed
Import-Module Pscx
$toto = "toto"
$test = ($toto,"-value=`"$toto`"")
Start-Process EchoArgs.exe -Wait -NoNewWindow -ArgumentList $test


Result by Powershell v2:

Arg 0 is <toto>
Arg 1 is <-value=toto>

The correct result should be:

Arg 0 is <toto>
Arg 1 is <-value="toto">
Posted by Oisín Grehan on 8/17/2010 at 7:51 AM
MSDeploy.exe (web deployment tool) is practically unusable in powershell because of the hellish nature of this problem.
Posted by Microsoft Connect on 1/29/2010 at 1:55 PM
Your bug has not been fixed in PowerShell 2.0. However we have kept the bug active for next version
Posted by Joel 'Jaykul' Bennett on 11/4/2009 at 4:35 PM
Check the workarounds, that "Invoke-Legacy" is called Start-Process ;-)
Posted by Charlie Russel - MVP on 11/4/2009 at 3:45 PM
Sorry, fat fingers gave me double quotes in that comment when it should have been single quotes. But still lame. ;)
Posted by Charlie Russel - MVP on 11/4/2009 at 3:41 PM
Interesting suggestion from Kirk. I like it. Especially the alias. ;)

My current workaround is:
cmd /c "whatever it is I need to invoke"

Which is really stupid and lame, but PoSH doesn't always make it easy to avoid this. If our goal is to get rid of the need for CMD, then we need a way to more transparently handle this sort of scenario.
Posted by Kirk Munro on 11/4/2009 at 2:01 PM
Maybe this could be solved in PowerShell v3 with a new cmdlet, Invoke-LegacyCommand, that takes a script block with whatever legacy commands you want to invoke. Then it could simply parse that script block with specific rules for legacy commands (maybe evaluating variables inside double-quotes but leaving the rest as is), invoking the legacy commands correctly without a lot of quoting horseplay.

And you could give it a default alias of "cmd".... :)