MASM default rules don't support 64-bit and actively *prevent* using 64-bit MASM

MASM default rules don't support 64-bit and actively *prevent* using 64-bit MASM by Bruce Dawson

Closed

as Fixed

Type:	Bug
ID:	371743
Opened:	10/1/2008 10:51:10 AM
Access Restriction:	Public

Workaround(s)

view

User(s) can reproduce this bug

When you add a .asm file to a VS project then it asks you if you want to use the default rule to run ML. This has the advantage of giving you a properties page for controlling ml.exe.

However it has the disadvantage of creating rules that don't support 64-bit compilation. The 32-bit version of ML.exe is invoked in all configurations, generating 32-bit code. There is no option in the properties page to invoke the 64-bit compiler.

Worse yet, once you've used this rule once in a project it is automatically invoked in the future. If you remove the .asm file and then add it back in it is automatically added with the masm.rules property page. At that point the only option you have for adding the .asm file and getting it work in a 64-bit build is to hand-edit the .vcproj file.

Details (expand)

English

Visual Studio 2008 Service Pack 1

Windows Vista

English

Create a C++ project. Create an x64 configuration. Add a .asm file and accept the default rule. Build for 32-bit all is well. Then build for 64-bit. Notice that the 32-bit assembler is invoked generating a 32-bit .obj file which leads to this linker error message:
1>Linking...
1>.\x64\Release\my.obj : fatal error LNK1112: module machine type 'X86' conflicts with target machine type 'x64'

The 32-bit ml.exe is invoked even on 64-bit configurations, generating a 32-bit .obj file.

The 64-bit ml64.exe should be invoked on 64-bit configurations.

File Name	Submitted By	Submitted On	File Size
Testx64.zip (restricted)	Bruce Dawson	10/1/2008	-

Thank you for contacting Microsoft.

On adding an ASM file to a VS project you will be prompted with a suggestion for a Custom Build Rule for the ASM file. As you have discovered these rules are specific to the x86 (32-bit) assembler. We are currently discussing how we want to improve this for the next release.

But for the current release, there is an edit you can make to the XML file installed with your copy of Visual Studio that contains the information for the ASM Custom Build Rule. As with any edit like this, be sure to save a copy of the original file to a safe place in the event you need to revert your change.

The file:

Microsoft Visual Studio 9.0\VC\VCProjectDefaults\masm.rules

contains one <CustomBuildRule ...> ... </CustomBuildRule> section. This is specific to the x86 (32-bit) assembler. The edit is to duplicate that section in its entirety (everything beginning with and including the line "<CustomBuildRule" through and including the line "</CustomBuildRule>") placing the duplicate immediately after the original. This will be done correctly if you have the line "</CustomBuildRule>" of the original section immediately followed by the "<CustomBuildRule" of the duplicated section on the next line (all contained within the "<Rules> .. </Rules>" node). Be sure that each "<CustomeBuildRule ... >" is closed with its own "</CustomBuildRule>".

The <CustomBuildRule ... > node of the duplicate will then require edits to two lines. The original looks like this:

        <CustomBuildRule
            Name="MASM"
            DisplayName="Microsoft Macro Assembler"
            CommandLine="ml.exe /c [AllOptions] [AdditionalOptions] /Ta[inputs]"
            Outputs="[$ObjectFileName]"
            FileExtensions="*.asm"
            ExecutionDescription="Assembling..."
            >

The two lines in the duplicated section should be edited to something like this (the 'DisplayName' line and the 'CommandLine' line)::

        <CustomBuildRule
            Name="MASM"
            DisplayName="Microsoft Macro Assembler for x64"
            CommandLine="ml64.exe /c [AllOptions] [AdditionalOptions] /Ta[inputs]"
            Outputs="[$ObjectFileName]"
            FileExtensions="*.asm"
            ExecutionDescription="Assembling..."
            >

With this edit, adding an ASM file to a VS project will prompt you with two choices as to which Custom Build Rule to use. This then gives you part of the solution.

The second part of the solution stems from the fact that ASM source is generally not portable between platforms. This means that you will need to add two ASM source files if you want to include ASM files in both x86 and x64 configurations of your project. Adding two different platform versions of your ASM source then exposes the issue that you can not assemble/link the x64 ASM file in the x86 configurations nor can you assemble/link the x86 ASM file in the x64 configurations.

You can solve this by excluding specific ASM files from specific configurations. You exclude the x64 ASM file from the x86 (Win32) Debug and Release configurations builds. And you exclude the x86 ASM file from the x64 Debug and Release configurations builds. While you are doing this you can check to make sure the correct set of Custom Build Rules are being use for the ASM files that remain in the various configurations.

I hope this enables you to proceed with you ASM development.

Thank you,
Visual Studio (and MASM) Product Team