Software building
description
Transcript of Software building
CHARLES UNIVERSITY IN PRAGUE
http://d3s.mff.cuni.cz
faculty of mathematics and physics
Software building
Tomáš Kalibera, Tomáš Pop
Tomáš Kalibera, Tomáš Pop
Plan
Next lesson
• SW building• Requirements• Typical scenarios• Dependency problems• ….
• Tools• Make• Ant• Make – advanced options• Autotools {conf, make, scan,..} • Ivy, Maven, SCons…
This lesson
Tomáš Kalibera, Tomáš Pop
• Transforming software sources (tree of source files) into executable binary code
• In general, it is simmilar to other transformations of files that are carried out by independent applications, such as Transforming LaTeX document into PDF Re-sizing and publishing a tree of photographs Running functional tests Updating a web site, translating XML source
documents into XHTML by XSLT transformer
What is (for this lecture) software building
Tomáš Kalibera, Tomáš Pop
Try to find some
Typical requirements on software building
Tomáš Kalibera, Tomáš Pop
• Automation Require no or minimal input from user Run regularly or based on source code
modifications• Portability
Run on all systems supported by the software Do not require (too many) additional tools,
compared to the software built Allow programmers to implement/configure
building easily for multiple platforms
Typical requirements on software building
Tomáš Kalibera, Tomáš Pop
• Efficiency Do not repeat what has already been done Employ multiple cores if available
• Robustness Build as much as possible – if there is any error,
try to build everything what is not affected by that error
Typical requirements on software building
Tomáš Kalibera, Tomáš Pop
• Generality The same tool should be usable for a wide range
of applications• Easy to use
Programmers – easy to define how building should be done, extensible
End users – easy to build a program with minimal (or ideally, without any) human intervention
Requirements on tools for software building
Tomáš Kalibera, Tomáš Pop
Use scenarios for build tools
Try describe typical scenarios (developer, end user)
Tomáš Kalibera, Tomáš Pop
• Usage pattern1. Download sources including the build
tool/configuration2. Run the tool to build everything from scratch3. If something goes wrong, fix the problem, delete
everything and start again
• Requirements Portability For simple programs, a (shell) script would probably
suffice Sometimes: include tests of resulting binary
One-shot build from scratch
Tomáš Kalibera, Tomáš Pop
• Usage pattern1. Build from scratch2. Modify some sources and/or configuration3. Build again, reusing previously built fragments
not affected by modifications• Requirements
Correctness – re-build everything what has to be re-built
Efficiency – do not re-build (much) more than that
Requirements on tools for software building
Tomáš Kalibera, Tomáš Pop
Problems to solved in build tools
Try to find some
Tomáš Kalibera, Tomáš Pop
• Building consists of running (many) external tools operating on files, in correct order
• Correct order must follow inter-dependencies of the source files i.e. first creating binary object files (.o), then the
library (.so, .a) Or, recompiling a source file (.c) when header it
depends on changes (.h)
Dependencies
Tomáš Kalibera, Tomáš Pop
• .h C-preprocessor header files• Pre-compiled header files• .c C source files• .o object files• .so, .a library files• Binary executables
preprocessor
preprocessor
compiler
archiver
linker
(Build) Dependencies in C Programs
Tomáš Kalibera, Tomáš Pop
• Dependencies can be detected by C preprocessor (typically via the compiler, i.e. gcc –M)
• Dependencies depend on preprocessor directives This typically maps to different dependencies for different
platforms Detectors therefore sometimes only detect a superset of
possible dependencies
#include <stdio.h>#ifdef __LINUX__
#include <unistd.h>#endif
Dependency detection – header deps
Tomáš Kalibera, Tomáš Pop
• Mission impossible The information is not explicitly there, developers
must provide it • Some reverse engineering is still possible, if
needed (legacy systems..) Debug information in binary object files (.o) may
include source files Object files in a static library (.a) may be listed by
their names In theory, listing symbols in executables might
help
Dependency detection – other C deps
Tomáš Kalibera, Tomáš Pop
• .java source files• .class bytecode files• .jar archiver files
compiler
archiver
This becomes much harder with native code….
(Build) Dependencies in Java Programs
Tomáš Kalibera, Tomáš Pop
• Dependencies can be detected by (some) Java compilers (i.e. Jikes)
• Dependencies can be circular This can only be resolved by Java compiler – files
forming a dependency cycle have to be compiled together
• Some of these dependencies can also be detected from compiled .class files
import java.util.HashSet;import org.sdmt.build.JavaDependencyDetector;
Dependency detection in Java (packages)
Tomáš Kalibera, Tomáš Pop
• which .class files are needed ?• JAR is in fact a ZIP file
Try jar -xf foo.jar or use WinZip• Detecting dependencies is easy
Listing the files Contains Classpath entry
which .class files are needed ?
Tomáš Kalibera, Tomáš Pop
• Assembly (EXE, dll) contains metadata Reference to other assemblies Used by JIT compiler
• The DLL Hell (Windows XP a Windows 2003 Server) Several versions of a same assembly can be installed at the same time on
one computer At app start time Windows looks for application manifest – describes
required versions of used assemblies (in application.exe.manifest file or in RT_MANIFEST resource embedded inside executable)
C# Assemblies
Tomáš Kalibera, Tomáš Pop
Other problems solved in build tools
Tomáš Kalibera, Tomáš Pop
• Support for downloading (sources, binary dependencies) HTTP, FTP, SCP, … e. g. Apache Ivy (next lesson)
• Support for version control systems Checkout/export Or even commit of the changes
Getting the sources
Tomáš Kalibera, Tomáš Pop
• Creating a software package Including metadata Including documentation Including binaries for a given system… Compression of package content
• Purpose of the packages Distribution to end users Deployment into other applications (plugins,
modules…), i.e. EJB, Eclipse plugins
Packaging
Tomáš Kalibera, Tomáš Pop
• Generating documentation• Running or creating tests• Bugs checking by code analysis• Calculating code complexity/good-design
metrics
And much more…
Tomáš Kalibera, Tomáš Pop
Existing tools
Tomáš Kalibera, Tomáš Pop
1970 (Unix)
2010 (Now)
1977 (Make)
1994 (Autotools)
2000 (Scons, Ant)
2003 (Maven)
Image source: http://commandline.org.uk/python/ohloh-and-popularity-programming-languages-free-and-open-source-software/
Existing tools - Time-line
Tomáš Kalibera, Tomáš Pop
• Make and its derivatives Originally by Stuart Feldman, Bell Labs, 1977 GNU Make originally by Richard Stallman and Roland
McGrath, 1993• GNU Build System (Autotools)
Autoconf – configuration detector Automake – makefile generator Libtool – library creation abstraction tool
• Scons Python Similar functionality as autotools.
The non-Java (mostly C) world
Tomáš Kalibera, Tomáš Pop
• Apache Ant James Duncan Davidson, 2000 (initially intended
only for Tomcat) Today de-facto standard for open source Java
projects• Maven
Apache Software Foundation Support for remote repositories with software
components to build• Continuum, Cruise Control
Tools for continuous building
The Java world
Tomáš Kalibera, Tomáš Pop
Make
Tomáš Kalibera, Tomáš Pop
• Expert system On input it gets a set of rules and a target to accomplish
(build) Based on the rules only, (not any specific algorithm for
the program), it finds a way to construct the target• Target
Something to be done/built, i.e. program binary, object file foo.o, distribution package
• Rules Descriptions that can be converted to dependencies
between two files and to commands that transform the files
Make overview
Tomáš Kalibera
• Support for parallel builds Make can automatically decide what can be built
in parallel (without any changes to earlier well written makefiles)
• Support for “building as much as possible” Make can be run in a mode where it compiles
everything that does not depend on a target that cannot be built (not stopping on first error it encounters)
Not specifying build algorithm is advantage
Tomáš Kalibera
• Make file Textual file with description of rules, commands are
command line utilities (compiler, shell commands, etc)
• Build tree Make decides on what has to be modified by examining
modification dates If file foo.o depends on foo.c, and foo.c is newer than
foo.o, foo.o has to be recompiled..
• Make tool Run “make” to build the default target
Make from user’s perspective
Tomáš Kalibera, Tomáš Pop
OBJ = ../ant/main.o parse.o
all: prog
prog: $(OBJ)$(CC) -o $@ $(OBJ)
parse.o: parse.c parse.h$(CC) -c parse.c
Example from Peter Miller’s article “Recursive Make Considered Harmful”
targetprerequisity
command
Make file example
Tomáš Kalibera, Tomáš Pop
• Built in variables (CC, FC, …)• Implicit rules
Typical compiler invocations, based on built-in variables (CC, CFLAGS, CPPFLAGS,…)
• User functions, conditionals, variables• Internal functions
File-based, text and path manipulation• Remaking make files
Make files themselves can be re-made by make This is very useful for automated generation of
dependencies, that can be described within the make file itself
List of (GNU) Make file features
Tomáš Kalibera
• Portability of make files Make is standardized by POSIX, but not all systems are
POSIX compliant… GNU Make runs everywhere Writing make files to run both on non-UNIX systems (such
as Windows) and UNIX is however a challenge• Inability to describe more complex dependencies
than between two files This matters i.e. for CORBA idl compiler that generates 2
files from a single one• Some argue that Make syntax is not user friendly
… and propose XML syntax that is much worse
Make limitations
Tomáš Kalibera, Tomáš Pop
• Especially with older make implementations, specifying all the dependencies by hand was too much work Implicit rules help Automake (and other makefile generators)
overcome this issue More later…
Make limitations
Tomáš Kalibera, Tomáš Pop
Apache Ant
Tomáš Kalibera, Tomáš Pop
• Java based Primarily for compilation of Java programs Wide support for programs and tools common in the Java
world (EJBs, JSPs, JUnit, …) Easily extensible in Java
• Main features Allows very portable scripting of building for Java
applications
• Weaknesses Syntax hard to read (XML) Limited extensibility in dependencies
Ant summary
Tomáš Kalibera
• Project Represents the application to be built
• Target A name for a goal required for building (compilation,
packaging, generating documentation) Can depend on other targets
• Task An executable code fragment (i.e. execution of Java
compiler)
• Property Name/value pair (i.e. for the base directory)
Ant definitions
Tomáš Kalibera, Tomáš Pop
<project name="MyProject" default="dist" basedir="."> <description> simple example build file </description>
<property name="src" location="src"/> <property name="build" location="build"/> <property name="dist" location="dist"/>
<target name="init"> <mkdir dir="${build}"/> </target>
<target name="compile" depends="init“ description="compile the source " > <javac srcdir="${src}" destdir="${build}"/> </target> …</project>
This is a shortened version of an example from Apache Ant documentation
Ant build file structure
Tomáš Kalibera, Tomáš Pop
• Compilation Javac, jspc, rmic, depend, …
• Archiving Bzip2, zip, gzip, jar, rpm, tar, cab, …
• Remote communication ftp, scp, sshexec, sending emails, …
• Testing JUnit
• Scripting support JavaScripts, Java 1.6 based, …
• File-based operations Platform independent manipulation with files and
directories
Ant tasks
Tomáš Kalibera
• Recompile everything from scratch Just run the compiler on all sources all the time
• Re-use class files compiled previously “depend” task, deletes .class files obsoleted by
modified sources Dependencies are read from class files Some dependencies are however not discovered
Source dependency resolution
Tomáš Kalibera, Tomáš Pop
• Ant http://ant.apache.org/manual/index.html
• Make http://www.gnu.org/software/make/manual/
make.pdf http://aegis.sourceforge.net/auug97.pdf
Links
Tomáš Kalibera, Tomáš Pop
• So far, so good…
Problems
Tomáš Kalibera, Tomáš Pop
• .h C-preprocessor header files• .F90 module sources• .mod compiled module info• .F90, .F fortran sources• .o object files• .so, .a library files• Binary executables
preprocessor
compiler
compiler
archiver
linker
Problems: 1) Build dependencies in Fortran
Tomáš Kalibera, Tomáš Pop
• C sources have headers Something not generated, but needed for compilation of
sources header dependencies influence what has to be
recompiled• Java sources have packages
Something generated by compiler from sources, needed for compiling other sources
package dependencies influence the order of compilation• Fortran sources have both
Headers and modulesHardly could be handled
by tools like Ant
Problems: 1) Build dependencies in Fortran
Tomáš Kalibera
• Make It is necessary to modify Makefile when changing
structure• Building too much• Not all dependencies covered
Problems: 2) Changes in source structure
Tomáš Kalibera, Tomáš Pop
• Idea and figures from paper: Petter Miller Recursive Make Considered Harmful Unix Word often using recursive makefiles Consider following structure:
MyApp Makefile App
Makefile main.c
Parse Makefile parse.c parse.h parse.y
Problems: 3) Recursive makefiles
MODULES = App Parse
all:
For dir in $(MODULES); do \
(cd $$dir; ${MAKE} all; \
done all: main.o
main.o: main.c\ ../Parse/parse.h
$(CC) -I../Parse -c main.c
all: prog
prog: parse.o ../App/main.o
$(CC) -o $@ $<
parse.o: parse.c parse.h
$(CC) -c parse.c
parse.c parse.h: parse.y
$(YACC) -d parse.y
Tomáš Kalibera, Tomáš Pop
• Q: What happen if parse.y is changed and incremental build (top level Makefile) is run?
MyApp Makefile App
Makefile main.c
Parse Makefile parse.c parse.h parse.y
MODULES = App Parse
all:
For dir in $(MODULES); do \
(cd $$dir; ${MAKE} all; \
done all: main.o
main.o: main.c\ ../Parse/parse.h
$(CC) -I../Parse -c main.c
all: prog
prog: parse.o ../App/main.o
$(CC) -o $@ $<
parse.o: parse.c parse.h
$(CC) -c parse.c
parse.c parse.h: parse.y
$(YACC) -d parse.y
Problems: 3) Recursive Makefiles