autotools-idl: technical explanation

autotools-idl technical explanation

You can post your questions and/or criticism to the mailing list.
The code excerpts here are from before the variables for customization of the generated sources were introduced. The current state of the changes to automake is the same in principal, only with some more parametrization which would not serve for a better explanation here.

General
Depending on a still non-existing header
The IDL dependencies themselves (idldep)
Unification of the output of different IDL compilers (idlfix)
IDL compilation Makefile rules (idl.am)
Changes to depout.m4
Changes to automake.in

General

Here are my priorities in writing the automake-idl patch:

Change as little as possible.
Delete as little as possible.
Add as little as possible.

These priorities are dictated by the fact that automake-idl is a patch. It must be easy to be merged with later releases of automake.

Depending on a still non-existing header

A tricky part with IDL is that it is an intermediate language. A single generated C++ source file might depend on another generated C++ header (besides the header with the same name) because of IDL inclusions. Prior to the first build the C++ dependencies are totally unknown - the generated sources don't exist at all! Let me demonstrate the problem with an example:

Foo.idl produces Foo.hpp and Foo.cpp (also FooServer.hpp and FooServer.cpp but they are not needed for the example)
FooBar.idl includes Foo.idl and produces FooBar.hpp and FooBar.cpp
BarFoo.idl is neither included by nor includes Foo.idl or FooBar.idl and produces BarFoo.hpp and BarFoo.cpp

Here is the dependency graph as seen by the developer:

Foo.idl	<--	FooBar.idl
^ \|		^ \|
Foo.hpp	<--	FooBar.hpp
^ \|		^ \|
Foo.cpp		FooBar.cpp

                  BarFoo.idl

                  ^

                    |

                  BarFoo.hpp

                  ^

                    |

                  BarFoo.cpp

Note that the horizontal dependencies are not known during the generation of Makefile.in and thus will not be known to make before the first build (i.e. before the dependencies are generated). Let's assume that make decides to build FooBar.o before Foo.o.

make sees that FooBar.cpp and FooBar.hpp must be built from FooBar.idl and invokes the IDL compiler.
make, still unaware that FooBar.hpp needs Foo.hpp, invokes the C++ compiler on FooBar.cpp. The compilation will fail, apparently.

The obvious solution is the following:

(*) All the objects of the generated C++ source files must depened on all the generated C++ header files. Thus, all IDL compilations will preceed all C++ compilations.

While this is good for the first build, it's not good for later builds (when make will be aware of the FooBar.hpp --> Foo.hpp dependency). Why should FooBar.cpp be recompiled if the totally unrelated BarFoo.idl has been touched? So, (*) should be in force only before a successful build. While detecting a successful first build is not impossible, it is also not reasonable. So, (*) should be in force for every C++ object individually and only before a successful C++ compilation (with the dependecies for that C++ object generated as a side effect).

The initial (*) dependencies for the generated stubs and skeletons are written to their respective .Po depfiles by config.status (see the explanation of depout.m4). E.g. for FooBar.idl:

.deps/FooBar.Po initially contains:

FooBar.$(OBJEXT): \
        Foo.hpp \
        FooServer.hpp \
        FooBar.hpp \
        FooBarServer.hpp \
        BarFoo.hpp \
        BarFooServer.hpp

.deps/FooBarServer.Po initially contains:

FooBarServer.$(OBJEXT): \
        Foo.hpp \
        FooServer.hpp \
        FooBar.hpp \
        FooBarServer.hpp \
        BarFoo.hpp \
        BarFooServer.hpp

Each of these files will be overwritten with the real dependencies by depcomp on the first compilation of the respective C++ source file.

The IDL dependencies themselves (idldep)

The dependencies between IDL files are created by idldep which uses the C++ compiler to compute them (actually, $(CXX) -E $(IDLCPPFLAGS) $(AM_IDLCPPFLAGS)). It is run just before the IDL compiler. For example, the dependencies for FooBar.idl would be written in a file .deps/FooBar.Pidl which would contain:

FooBar.hpp FooBarServer.hpp FooBar.cpp FooBarServer.cpp: FooBar.idl Foo.idl
FooBar.idl:
Foo.idl:

Have in mind that not all depmodes are supported. If you manage to complement idldep with other depmodes, please submit your patches to the mailing list.

Unification of the output of different IDL compilers (idlfix)

The unification of the output of different IDL compilers is done by idlfix. It is run just after the IDL compiler.
Generally, IDL compilers write the generated sources in the current directory (some of them have an option for different output directory but they are not used). So, to support automake's subdir-objects option, the generated sources must be moved to the subdirectory after content unification.

IDL compilation Makefile rules (idl.am)

idl.am contains the patterns of Makefile rules needed to compile and IDL file (equivalent to depend2.am). There are three steps (in this order):

Dependencies (idldep invocation)
IDL compilation
Unification and subdir-objects support (idlfix)

It is important to note that only non-generic rules are used. Generic rules are not suitable for IDL and parallel builds. If generic rules are used then make would launch up to 4 (the number of the target files) compilations for the same IDL file at the same time, and this cannot be avoided (make assumes that the commands for the 4 targets are independent while in our case they are one and the same).

Changes to depout.m4

+  IDL_BASES=`sed -n -e '
+    /^IDL_BASES = .*\\\\$/ {
+      s/^IDL_BASES = //
+      :loop
+        s/\\\\$//
+        p
+        n
+        /\\\\$/ b loop
+      p
+    }
+    /^IDL_BASES = / s/^IDL_BASES = //p' < "${mf}"`
+    
+  GENERATED_HEADERS=
+  for base in ${IDL_BASES}
+  do
+      GENERATED_HEADERS="${GENERATED_HEADERS} ${base}.hpp ${base}Server.hpp"
+  done
+  
+  for base in ${IDL_BASES}
+  do
+      IDL_DEP_BASE=`echo "${base}" | sed 's|[[^\\/]]*$|'"${DEPDIR}"'/&|'`
+      echo ${ECHO_N} "${base}.\$(OBJEXT):${ECHO_C}" > "${dirpart}/${IDL_DEP_BASE}.Po"
+      echo ${ECHO_N} "${base}Server.\$(OBJEXT):${ECHO_C}" > "${dirpart}/${IDL_DEP_BASE}Server.Po"
+      for generated_header in ${GENERATED_HEADERS}
+      do
+          echo " \\" >> "${dirpart}/${IDL_DEP_BASE}.Po"
+          echo " \\" >> "${dirpart}/${IDL_DEP_BASE}Server.Po"
+          echo ${ECHO_N} "	${generated_header}${ECHO_C}" >> "${dirpart}/${IDL_DEP_BASE}.Po"
+          echo ${ECHO_N} "	${generated_header}${ECHO_C}" >> "${dirpart}/${IDL_DEP_BASE}Server.Po"
+      done
+      echo >> "${dirpart}/${IDL_DEP_BASE}.Po"
+      echo >> "${dirpart}/${IDL_DEP_BASE}Server.Po"
+  done

This snippet generates the (*) dependencies for the IDL-generated C++ sources. Note that it is executed after the depfiles are filled with "# dummy" by the original automake code.
1. The IDL bases are retrieved from Makefile.in. The method used is borrowed from automake 1.8.
2. The filenames of the generated stub and skeleton headers are collected in a single variable, so that they can be used later in a for loop.
3. For every IDL base, two .Po files are output - one for the stub and one for the skeleton C++ source file. Each of them contains a single make dependency of the corresponding object to all the generated headers. Every generated header's name is put on a separate line because many makes have a limit on the line length.
Note that the macros in depout.m4 are included in config.status and are executed only if dependencies are enabled.

Changes to automake.in

This is an explanation of the non-obvious parts of the current automake-idl patch. Order is not preserved.

+my
      %idl_bases;

This variable contains the names of the bases for the IDL files in the current Makefile.am.

+ %idl_bases = ();

Initializes the %idl_bases, once per Makefile.am.

+sub lang_idl_target_hook
+{
+    my ($self, $aggregate, $output, $input) = @_;
+    my $base;
+
+    if (! option 'no-dependencies')
+    {
+        my $depfile = $output;
+        $depfile =~ s/\.([^.]*)$/.Pidl/;
+        $dep_files{dirname ($depfile) . '/$(DEPDIR)/' . basename ($depfile)} = 1;
+    }
+
+    ($base = $output) =~ s/\.([^.]*)$//;
+    $idl_bases{$base} = 1;
+
+    $clean_files{$base . ".hpp"} = MOSTLY_CLEAN;
+    $clean_files{$base . "Server.hpp"} = MOSTLY_CLEAN;
+    $clean_files{$base . ".cpp"} = MOSTLY_CLEAN;
+    $clean_files{$base . "Server.cpp"} = MOSTLY_CLEAN;
+}

lang_idl_target_hook (called once per IDL file):
1. Registers the depfile (since handle_single_transform() wouldn't do this for an intermediate language).
2. Adds the base of the current IDL file to %idl_bases. Note that the value is obtained from the output rather than the input (using the input to generate the base in case there are subdir sources, but subdir-objects is not specified, would be a bug).
3. Adds the generated sources to the mostly-clean target.

+sub lang_idl_finish
+{
+    return if defined $language_scratch{'idl-done'};
+    $language_scratch{'idl-done'} = 1;
+
+    my @generated_idl_headers = ();
+    foreach my $base (keys %idl_bases)
+    {
+        push (@generated_idl_headers, $base . ".hpp");
+        push (@generated_idl_headers, $base . "Server.hpp");
+        if (! option 'no-dependencies')
+        {
+            Automake::Variable::define ('IDL_BASES', VAR_AUTOMAKE, '+', TRUE, $base, '', INTERNAL, VAR_PRETTY);
+        }
+    }
+
+    if (option 'no-dependencies')
+    {
+        foreach my $base (keys %idl_bases)
+        {
+            pretty_print_rule($base . ".\$$OBJEXT$ " . $base . "Server.\$$OBJEXT$:", "\t", @generated_idl_headers);
+        }
+    }
+    else
+    {
+        foreach my $base (keys %idl_bases)
+        {
+            pretty_print_rule("\@AMDEP_FALSE\@" . $base . ".\$$OBJEXT$ " . $base . "Server.\$$OBJEXT$:", "\@AMDEP_FALSE\@\t", @generated_idl_headers);
+        }
+    }
+}

lang_idl_finish (called once per Makefile.am):
1. Dumps %idl_bases to a Makefile.in variable $IDL_BASES (only in case dependencies are not turned off). $IDL_BASES would be used by depout.m4 to generate the rules (*).
2. In case dependencies are turned off, outputs the (*) rules unconditionally in Makefile.in.
3. In case dependencies are not turned off, outputs the (*) rules conditionally in Makefile.in (they will be in effect if the dependencies are turned off by configure's --disable-dependency-tracking option).

+    if ($extension_seen{'.idl'})
+    {
+        # Set location of idlfix.
+        define_variable ('idlfix',
+                    "\$(SHELL) $am_config_aux_dir/idlfix",
+                    INTERNAL);
+        require_conf_file ("$am_file.am", FOREIGN, 'idlfix');
+    }

Installs idlfix if automake has been called with the -a option.

Installs idldep if automake has been called with the -a option.

             if ($have_per_exec_flags)
             {
+                err_am ($lang->Name . " does not support per target compiler flags")
+                  if $lang->name eq "idl";

IDL cannot support per target compiler flags (not easily, at least) because the IDL compilers do not have options to customize the prefix of the names of the generated files. Simply renaming them (e.g. in idlfix) is not an option - the #include directives should be modified, too.

+            # IDL requires explicit rules in any case.
+            if ($lang->name eq "idl")
+            {
+                $renamed = 1;
+            }

This snippet forces automake not to try to use generic rules for IDL files. See the explanation of idl.am.

        if ($derived_source)
          {
-           prog_error ($lang->name . " has automatic dependency tracking")
-             if $lang->autodep ne 'no';

I really don't see a reason why an intermediate language cannot have automatic dependency tracking.

            push_dist_common ($object)
-             unless ($topparent =~ /^(?:nobase_)?nodist_/);
+              unless ($topparent =~ /^(?:nobase_)?nodist_/) ||
+              (defined $clean_files{$object} && ($clean_files{$object} eq CLEAN || $clean_files{$object} eq MOSTLY_CLEAN));
            next;

IMHO, the ones added to the clean and mostly-clean targets shouldn't be distributed, too.

       foreach my $dest (&{$lang->output_extensions} ($suffix))
        {
          register_suffix_rule (INTERNAL, $suffix, $dest);
+          accept_extensions ($dest);
        }

See bug report 424 on the automake Gnats database.