CHARLES UNIVERSITY IN PRAGUE

http://d3s.mff.cuni.cz

faculty of mathematics and physics

Software building

Tomáš Kalibera, Tomáš Pop

Tomáš Kalibera, Tomáš Pop

Plan

Next lesson

• SW building• Requirements• Typical scenarios• Dependency problems• ….

• Tools• Make• Ant• Make – advanced options• Autotools {conf, make, scan,..} • Ivy, Maven, SCons…

This lesson

Tomáš Kalibera, Tomáš Pop

• Transforming software sources (tree of source files) into executable binary code

• In general, it is simmilar to other transformations of files that are carried out by independent applications, such as Transforming LaTeX document into PDF Re-sizing and publishing a tree of photographs Running functional tests Updating a web site, translating XML source

documents into XHTML by XSLT transformer

What is (for this lecture) software building

Tomáš Kalibera, Tomáš Pop

Try to find some

Typical requirements on software building

Tomáš Kalibera, Tomáš Pop

• Automation Require no or minimal input from user Run regularly or based on source code

modifications• Portability

Run on all systems supported by the software Do not require (too many) additional tools,

compared to the software built Allow programmers to implement/configure

building easily for multiple platforms

Typical requirements on software building

Tomáš Kalibera, Tomáš Pop

• Efficiency Do not repeat what has already been done Employ multiple cores if available

• Robustness Build as much as possible – if there is any error,

try to build everything what is not affected by that error

Typical requirements on software building

Tomáš Kalibera, Tomáš Pop

• Generality The same tool should be usable for a wide range

of applications• Easy to use

Programmers – easy to define how building should be done, extensible

End users – easy to build a program with minimal (or ideally, without any) human intervention

Requirements on tools for software building

Tomáš Kalibera, Tomáš Pop

Use scenarios for build tools

Try describe typical scenarios (developer, end user)

Tomáš Kalibera, Tomáš Pop

• Usage pattern1. Download sources including the build

tool/configuration2. Run the tool to build everything from scratch3. If something goes wrong, fix the problem, delete

everything and start again

• Requirements Portability For simple programs, a (shell) script would probably

suffice Sometimes: include tests of resulting binary

One-shot build from scratch

Tomáš Kalibera, Tomáš Pop

• Usage pattern1. Build from scratch2. Modify some sources and/or configuration3. Build again, reusing previously built fragments

not affected by modifications• Requirements

Correctness – re-build everything what has to be re-built

Efficiency – do not re-build (much) more than that

Requirements on tools for software building

Tomáš Kalibera, Tomáš Pop

Problems to solved in build tools

Try to find some

Tomáš Kalibera, Tomáš Pop

• Building consists of running (many) external tools operating on files, in correct order

• Correct order must follow inter-dependencies of the source files i.e. first creating binary object files (.o), then the

library (.so, .a) Or, recompiling a source file (.c) when header it

depends on changes (.h)

Dependencies

Tomáš Kalibera, Tomáš Pop

• .h C-preprocessor header files• Pre-compiled header files• .c C source files• .o object files• .so, .a library files• Binary executables

preprocessor

preprocessor

compiler

archiver

linker

(Build) Dependencies in C Programs

Tomáš Kalibera, Tomáš Pop

• Dependencies can be detected by C preprocessor (typically via the compiler, i.e. gcc –M)

• Dependencies depend on preprocessor directives This typically maps to different dependencies for different

platforms Detectors therefore sometimes only detect a superset of

possible dependencies

#include <stdio.h>#ifdef __LINUX__

#include <unistd.h>#endif

Dependency detection – header deps

Tomáš Kalibera, Tomáš Pop

• Mission impossible The information is not explicitly there, developers

must provide it • Some reverse engineering is still possible, if

needed (legacy systems..) Debug information in binary object files (.o) may

include source files Object files in a static library (.a) may be listed by

their names In theory, listing symbols in executables might

help

Dependency detection – other C deps

Tomáš Kalibera, Tomáš Pop

• .java source files• .class bytecode files• .jar archiver files

compiler

archiver

This becomes much harder with native code….

(Build) Dependencies in Java Programs

Tomáš Kalibera, Tomáš Pop

• Dependencies can be detected by (some) Java compilers (i.e. Jikes)

• Dependencies can be circular This can only be resolved by Java compiler – files

forming a dependency cycle have to be compiled together

• Some of these dependencies can also be detected from compiled .class files

import java.util.HashSet;import org.sdmt.build.JavaDependencyDetector;

Dependency detection in Java (packages)

Tomáš Kalibera, Tomáš Pop

• which .class files are needed ?• JAR is in fact a ZIP file

Try jar -xf foo.jar or use WinZip• Detecting dependencies is easy

Listing the files Contains Classpath entry

which .class files are needed ?

Tomáš Kalibera, Tomáš Pop

• Assembly (EXE, dll) contains metadata Reference to other assemblies Used by JIT compiler

• The DLL Hell (Windows XP a Windows 2003 Server) Several versions of a same assembly can be installed at the same time on

one computer At app start time Windows looks for application manifest – describes

required versions of used assemblies (in application.exe.manifest file or in RT_MANIFEST resource embedded inside executable)

C# Assemblies

Tomáš Kalibera, Tomáš Pop

Other problems solved in build tools

Tomáš Kalibera, Tomáš Pop

• Support for downloading (sources, binary dependencies) HTTP, FTP, SCP, … e. g. Apache Ivy (next lesson)

• Support for version control systems Checkout/export Or even commit of the changes

Getting the sources

Tomáš Kalibera, Tomáš Pop

• Creating a software package Including metadata Including documentation Including binaries for a given system… Compression of package content

• Purpose of the packages Distribution to end users Deployment into other applications (plugins,

modules…), i.e. EJB, Eclipse plugins

Packaging

Tomáš Kalibera, Tomáš Pop

• Generating documentation• Running or creating tests• Bugs checking by code analysis• Calculating code complexity/good-design

metrics

And much more…

Tomáš Kalibera, Tomáš Pop

Existing tools

Tomáš Kalibera, Tomáš Pop

1970 (Unix)

2010 (Now)

1977 (Make)

1994 (Autotools)

2000 (Scons, Ant)

2003 (Maven)

Image source: http://commandline.org.uk/python/ohloh-and-popularity-programming-languages-free-and-open-source-software/

Existing tools - Time-line

Tomáš Kalibera, Tomáš Pop

• Make and its derivatives Originally by Stuart Feldman, Bell Labs, 1977 GNU Make originally by Richard Stallman and Roland

McGrath, 1993• GNU Build System (Autotools)

Autoconf – configuration detector Automake – makefile generator Libtool – library creation abstraction tool

• Scons Python Similar functionality as autotools.

The non-Java (mostly C) world

Tomáš Kalibera, Tomáš Pop

• Apache Ant James Duncan Davidson, 2000 (initially intended

only for Tomcat) Today de-facto standard for open source Java

projects• Maven

Apache Software Foundation Support for remote repositories with software

components to build• Continuum, Cruise Control

Tools for continuous building

The Java world

Tomáš Kalibera, Tomáš Pop

Make

Tomáš Kalibera, Tomáš Pop

• Expert system On input it gets a set of rules and a target to accomplish

(build) Based on the rules only, (not any specific algorithm for

the program), it finds a way to construct the target• Target

Something to be done/built, i.e. program binary, object file foo.o, distribution package

• Rules Descriptions that can be converted to dependencies

between two files and to commands that transform the files

Make overview

Tomáš Kalibera

• Support for parallel builds Make can automatically decide what can be built

in parallel (without any changes to earlier well written makefiles)

• Support for “building as much as possible” Make can be run in a mode where it compiles

everything that does not depend on a target that cannot be built (not stopping on first error it encounters)

Not specifying build algorithm is advantage

Tomáš Kalibera

• Make file Textual file with description of rules, commands are

command line utilities (compiler, shell commands, etc)

• Build tree Make decides on what has to be modified by examining

modification dates If file foo.o depends on foo.c, and foo.c is newer than

foo.o, foo.o has to be recompiled..

• Make tool Run “make” to build the default target

Make from user’s perspective

Tomáš Kalibera, Tomáš Pop

OBJ = ../ant/main.o parse.o

all: prog

prog: $(OBJ)$(CC) -o $@ $(OBJ)

parse.o: parse.c parse.h$(CC) -c parse.c

Example from Peter Miller’s article “Recursive Make Considered Harmful”

targetprerequisity

command

Make file example

Tomáš Kalibera, Tomáš Pop

• Built in variables (CC, FC, …)• Implicit rules

Typical compiler invocations, based on built-in variables (CC, CFLAGS, CPPFLAGS,…)

• User functions, conditionals, variables• Internal functions

File-based, text and path manipulation• Remaking make files

Make files themselves can be re-made by make This is very useful for automated generation of

dependencies, that can be described within the make file itself

List of (GNU) Make file features

Tomáš Kalibera

• Portability of make files Make is standardized by POSIX, but not all systems are

POSIX compliant… GNU Make runs everywhere Writing make files to run both on non-UNIX systems (such

as Windows) and UNIX is however a challenge• Inability to describe more complex dependencies

than between two files This matters i.e. for CORBA idl compiler that generates 2

files from a single one• Some argue that Make syntax is not user friendly

… and propose XML syntax that is much worse

Make limitations

Tomáš Kalibera, Tomáš Pop

• Especially with older make implementations, specifying all the dependencies by hand was too much work Implicit rules help Automake (and other makefile generators)

overcome this issue More later…

Make limitations

Tomáš Kalibera, Tomáš Pop

Apache Ant

Tomáš Kalibera, Tomáš Pop

• Java based Primarily for compilation of Java programs Wide support for programs and tools common in the Java

world (EJBs, JSPs, JUnit, …) Easily extensible in Java

• Main features Allows very portable scripting of building for Java

applications

• Weaknesses Syntax hard to read (XML) Limited extensibility in dependencies

Ant summary

Tomáš Kalibera

• Project Represents the application to be built

• Target A name for a goal required for building (compilation,

packaging, generating documentation) Can depend on other targets

• Task An executable code fragment (i.e. execution of Java

compiler)

• Property Name/value pair (i.e. for the base directory)

Ant definitions

Tomáš Kalibera, Tomáš Pop

<project name="MyProject" default="dist" basedir="."> <description> simple example build file </description>

<property name="src" location="src"/> <property name="build" location="build"/> <property name="dist" location="dist"/>

<target name="init"> <mkdir dir="${build}"/> </target>

<target name="compile" depends="init“ description="compile the source " > <javac srcdir="${src}" destdir="${build}"/> </target> …</project>

This is a shortened version of an example from Apache Ant documentation

Ant build file structure

Tomáš Kalibera, Tomáš Pop

• Compilation Javac, jspc, rmic, depend, …

• Archiving Bzip2, zip, gzip, jar, rpm, tar, cab, …

• Remote communication ftp, scp, sshexec, sending emails, …

• Testing JUnit

• Scripting support JavaScripts, Java 1.6 based, …

• File-based operations Platform independent manipulation with files and

directories

Ant tasks

Tomáš Kalibera

• Recompile everything from scratch Just run the compiler on all sources all the time

• Re-use class files compiled previously “depend” task, deletes .class files obsoleted by

modified sources Dependencies are read from class files Some dependencies are however not discovered

Source dependency resolution

Tomáš Kalibera, Tomáš Pop

• Ant http://ant.apache.org/manual/index.html

• Make http://www.gnu.org/software/make/manual/

make.pdf http://aegis.sourceforge.net/auug97.pdf

Links

Tomáš Kalibera, Tomáš Pop

• So far, so good…

Problems

Tomáš Kalibera, Tomáš Pop

• .h C-preprocessor header files• .F90 module sources• .mod compiled module info• .F90, .F fortran sources• .o object files• .so, .a library files• Binary executables

preprocessor

compiler

compiler

archiver

linker

Problems: 1) Build dependencies in Fortran

Tomáš Kalibera, Tomáš Pop

• C sources have headers Something not generated, but needed for compilation of

sources header dependencies influence what has to be

recompiled• Java sources have packages

Something generated by compiler from sources, needed for compiling other sources

package dependencies influence the order of compilation• Fortran sources have both

Headers and modulesHardly could be handled

by tools like Ant

Problems: 1) Build dependencies in Fortran

Tomáš Kalibera

• Make It is necessary to modify Makefile when changing

structure• Building too much• Not all dependencies covered

Problems: 2) Changes in source structure

Tomáš Kalibera, Tomáš Pop

• Idea and figures from paper: Petter Miller Recursive Make Considered Harmful Unix Word often using recursive makefiles Consider following structure:

MyApp Makefile App

Makefile main.c

Parse Makefile parse.c parse.h parse.y

Problems: 3) Recursive makefiles

MODULES = App Parse

all:

For dir in $(MODULES); do \

(cd $$dir; ${MAKE} all; \

done all: main.o

main.o: main.c\ ../Parse/parse.h

$(CC) -I../Parse -c main.c

all: prog

prog: parse.o ../App/main.o

$(CC) -o $@ $<

parse.o: parse.c parse.h

$(CC) -c parse.c

parse.c parse.h: parse.y

$(YACC) -d parse.y

Tomáš Kalibera, Tomáš Pop

• Q: What happen if parse.y is changed and incremental build (top level Makefile) is run?

MyApp Makefile App

Makefile main.c

Parse Makefile parse.c parse.h parse.y

MODULES = App Parse

all:

For dir in $(MODULES); do \

(cd $$dir; ${MAKE} all; \

done all: main.o

main.o: main.c\ ../Parse/parse.h

$(CC) -I../Parse -c main.c

all: prog

prog: parse.o ../App/main.o

$(CC) -o $@ $<

parse.o: parse.c parse.h

$(CC) -c parse.c

parse.c parse.h: parse.y

$(YACC) -d parse.y

Problems: 3) Recursive Makefiles