Framewave Build System

 

Architecture

 

 

 

Introduction

 

Building Framewave requires some features that are currently not provided by any existing build system publicly available. Driven by this, we wrote a custom build framework, based on the SCons build tool.

 

This build system currently works on four different operating systems, using three different toolsets. It provides features to accomplish all the requirements that have come up in the last six months of its existence.

 

Using SCons gives it access to some very useful features, primarily,

·         Having a full scale language available to write the build scripts in

·         SCons’s dependency scanning system, which works on actual changes to files versus time stamps

·         SCon’s ability to do efficient parallel builds

 

 

Prerequisites

 

·         Working knowledge of SCons. We are currently using version 0.97. Scons documentation is available from, http://www.scons.org

·         Working knowledge of Python

·         Working knowledge of SIMD programming from C++

·         Expert knowledge of compilers and linkers

 

This document can be ‘understood’ without the above prerequisites but to be able truly grasp how this build system works, they are essential.

 

Requirements

 

The initial requirement for this build system was to accomplish a multi-pass compile. A multi-pass compile would be described as:

 

The ability to compile each source file ‘n’ number of times, where ‘n’ is the number of different optimization paths implemented in the source file. The ‘n’ different object files are then linked into one final binary. Also, a stub function is generated, compiled, and linked into this binary. The stub functions’ purpose is to pose as the actual exported function, check the dispatch type of the current processor, and then execute the correct function in the appropriate object file.

 

So if we have an exported function called fwAdd_32f, which had different implementations for reference code, SSE2 optimized code and Family 10h optimized code, we would compile the file containing fwAdd_32f three time; during each compilations, a prefix is added to fwAdd_32f function and an object file is produced.. We would then compile a stub for fwAdd_32f which would call the appropriate version of the original fwAdd_32f function based on which processor we were running on. Then we would link all of these three object files and the object file with the stub function, into one final binary, which could be a shared library, static library or an executable.

 

Considering the complexity of this task, it made sense to have ‘one’ build system in place that would do this for all our current operating systems (back then only Windows and Linux) and toolsets (msvc and gcc). With these motivations, the build work was started and over time a large number of requirements were added.

 

As of now, the current list of requirements stand at;

 

·         Perform a multi-pass compile for all files designated for it

o        For files we do not want compiled multi-pass, the build system should still be able to compile them as they would have been under a regular single pass build system

·         Work with largely the same code base on all supported operating systems and for all supported toolsets

o        We currently support builds for,

§         Windows

§         Linux

§         Solaris

§         Mac (Darwin)

o        We can currently build with,

§         MSVC

§         GCC

§         Sun CC

·         Provide the ability to use flags based on any of the following conditions

o        Compiler (implicitly)

o        Operating System

o        Variant

o        Bitness

o        Library type

o        Optimization path compiled for

o        Library compiled for

o        Filename

·         Additionally, provide the ability to give exceptions in either compilation or linking based on the same set of conditions

o        These exceptions can range from not building a particular file at all, to using a completely different set of flags for a particular build of a particular file

·         Provide incremental builds based on changes in the requisite dependency chains

·         Provide parallel builds

·         Provide the ability to automatically pick up CPP files and Header files in the source and include directories designated by the project directory structure (plus header files from the common includes)

o        The idea behind providing the includes was to not have to give the path for a header file, ever; which were starting to get long, complicated and made it difficult to move things around

§         As a side effect of this feature, we cannot have two header files of the same name, unless we specify the relative or absolute path to it (in essence, don’t use this feature)

·         Provide command line support for

o        What to build

o        Which variant to build

o        Which library type to build

o        Which bitness to build

o        Provide arbitrary flags

o        Provide access to special services like,

§         Providing debug info with release builds

§         Linking with different CRT models on MSVC

 

 

Organization

 

SCons requires us to provide it with a main SConstruct file for a base project and .sconscript files for each sub project. We currently have SConstructs for the Framewave and Test projects.

 

The projects are organized such;

 

·         Framewave\trunk

o        FW_Sourceforge\Framewave      [Sconstruct here]

§         domain\fwBase           [sconscript here]

§         domain\fwImage          [sconscript here]

§         domain\fwJPEG           [sconscript here]

§         domain\fwSignal         [sconscript here]

§         domain\fwVideo          [sconscript here]

o        FW_AMD\Test                   [Sconstruct here]

§         G3                      [sconscript here]

§         TestLib                 [Not under SCons yet]

§         UnitTests               [Not under SCons yet-]

[-No plans as of now]

o        BuildTools\buildscripts

§         Our build system lives here

 

 

Usage

 

For the build system to function correctly, the paths to the build tools to be used need to be setup before the invocation of SCons

 

To build, we need to invoke SCons in either of the directories that contain the SConstructs. The following command line parameters are supported;

 

subProjectName

variant=[debug|release]

libtype=[shared|static]

bitness=[32|64]

 

If any of these are not provided, the default value (in bold) would be used. If no subproject is provided, all existing subprojects would be built.

 

The following parameters have default values that will be picked on multiple different build factors.

debuginfo=[on|off]

wincrt   =[mt|mtd|md|mdd]

toolset  =[msvc|gcc|suncc]

 

Along with these, we can specify a target name on the command line to build only that one project. For example;

 

scons fwImage

 

will build only fwImage

 

High Level Design

 

The purpose of the build system is to setup a hierarchy of SCons objects, ending in top level targets. The top level target would either be a library or an executable.

 

We provide specific flags for every object that we setup.

 

We accomplish this by providing our own factory objects for both object files and library files. These factory objects return an SCons object with all the flags correctly set up.

 

Taking a basic example of the fwSignal shared library on Windows, compiling only with Add.cpp, this is the SCons object hierarchy we finally end up with at the end of the execution of our scripts.

 

 

Organization Chart

 

 

The three object files with the prefixes “_refr_”, “_sse2” and “_f10h” are the multi-pass compiled object files generated from Add.cpp. The fwSignal_opt.obj file is the file containing the stub functions for all the exported functions from Add.cpp. fwSignal.obj is the object file generated from compiling fwSignal.cpp with a single pass compilation.

 

High Level Flow of Control (FOC)

 

 

FoC in the SConstruct file:

 

Our execution begins in the SConstruct file and then goes onto the sconscript files. This transition is not controlled by us, we simply register which sconscript files are needed by this project and SCons takes the onus of calling them. The high level execution of our SCons script files is as follows;

 

1.       Create an fwBuildRoot object. This object at the moment is always called “oRoot”

2.       Create the fwBase subproject from the oRoot object

a.       This calls into the subProject method of the oRoot in fwbuild.py. This function registers the sconscript for the subproject, at which point SCons transfers execution to the given sconscript file and then returns control to our script

3.       Iterate through the list of subprojects

a.       Create subprojects from the oRoot object for all

                                                               i.      For each subproject, as we register the requisite sconscript, SCons executes that sconscript and then returns control to our script

 

The subProject method in oRoot returns a Library or Program build object, which is a target a SCons.

 

We then issue an installation directive to SCons for this object and create an alias for the object returned by that directive which maps to just the name of the library the SCons would be building.

At that point our execution of the SConscript (and of course, any and all of our build scripts) is over. SCons takes over and builds the libraries using the hierarchy and flags that we have specified.

 

Dependency checking and parallel builds are handled by SCons based on the object, operating system, toolset and flags that we have setup.

 

 

FoC in a sconscript file:

 

Once the sconscript is executed, we create a fwProject object for the subproject. Then we call either the initBuildObjects or initMultipassBuildObjects method of the fwProject object. These methods are also present in the fwbuild.py file.

 

FoC in the initBuildObjects method:

 

The initBuildObjects method gets the list of CPP files and finally returns a SCons Library or Program object back to the SConscript file. This is its flow of control:

 

1.       Get a list of all the CPP files we need to build

2.       Call the constructObjects to create all the requisite object files. The function return a list of SCons SharedObject or StaticObject objects to us

3.       Create an fwLibrary factory object, passing this list of objects as it’s dependencies

4.       Return the SCons Library object from the fwLibrary object

 

FoC in the initMultipassBuildObjects function:

 

The initMultipassBuildObjects method gets the list of CPP files and finally returns a SCons Library or Program object back to the SConscript file. During this process it also builds the stub CPP and processes any files given to it in its exclude list with single pass processing. This is its flow of control:

 

1.       Get a list of all the CPP files we need to build

2.       Create the stub CPP file using a call to constructMultipassCPP [which exists in fwparse.py]

3.       Extract a list of exclude files, which includes,

a.       Any files given in the exclude list parameter

b.       The fwProjectName.cpp file

c.       The created stub CPP file

4.       Call constructMultipassObjects to create all the requisite object files. The functions return a list of SCons SharedObject or StaticObject objects

5.       Call constructObjects on the list of exclude files to generate object files for the single pass CPP files

6.       Add both the lists together into one final list of object files

7.       Create an fwLibrary factory object, passing this list of objects as it’s dependencies

8.       Return the SCons Library object from the fwLibrary object

 

 

FoC in the constructObjects function:

 

constructObjects returns a list of SCons SharedObject or StaticObject objects, based on the list of CPP files that it gets as an input. The flow of control is as follows:

 

1.       From the CPP filename, get the Object filename

2.       For each CPP file;

a.       Create a fwObject factory object, passing it the object filename and CPP filename

b.       Get the SCons SharedObject or StaticObject and add it to the list of objects

3.       Return the list of objects

 

FoC in the constructMultipassObjects function:

 

1.       From the CPP filename, get the Object filename

2.       For each CPP file;

a.       Create a fwMPObject factory object, passing it the object filename and CPP filename

b.       Get the list of SCons SharedObject or StaticObject objects and add it to the list of objects

3.       Return the list of objects

 

The functionality of the fwObject, fwMPObject and fwLibrary object factory classes will be discussed in their respective low level design documents

 

 

Issues

 

 

The sub components of the build system have been completely thrown out and redesigned to meet the current list of requirements. The fwbuild.py file, though fairly changed around, has been relatively redesigned less. This code has a lot of legacy features. This section lists all the know ones for now. To find all the features that either need to be fixed or enhanced, search through the build code for the strings “BUGBUG:” or “TODO:” with that exact case.