Application Installation

Guntis BārzdiņšĢirts Folkmanis

Artūrs LavrenovsNormunds Grūzītis

“Hello World”

$ cat > hello.c#include <stdio.h>int main() { printf("Hello World!\n"); return 0;}Ctrl/D$ gcc hello.c$ ./a.outHello World!$

GCC: the GNU Compiler Collection

Originally: GNU C Compiler (1987) Frontends for:

C: gcc (vs. GCC) C++: g++ More (ada, java, objective-c, fortran, …)

Backends for: x86, ia-64, ppc, m68k, alpha, hppa, mips, sparc, mmix, pdp-11, vax, …

GCC: the GNU Compiler Collection.c .java ...

gcc gcj ...

AST

ASM

OBJ

EXE

Compiling C programs

Involves at least four kinds of files: Regular source code files (*.c) Header files (*.h) Object files (*.o) Binary executables (*.out)

Other kinds of files: Libraries (*.a) Shared libraries (*.so)

gcc: Usage

gcc foo.cpp Outputs a.out executable binary

gcc -Wall -pedantic foo.cpp bar.cpp baz.o -o foo -Wall: warns about legal but dubious code constructs, helps to catch

a lot of bugs early -pedantic: warns about non-portable (non-standard) constructs -o: outputs the executable to the specified file

The standard C library (libc) is linked by default To include other libraries, e.g. the math library: -lm

The static library m is located in libm.a

Steps involved

1. Preprocessor (cpp)

2. Compiler (gcc)

3. Assembler (as)

4. Linker (ld)

Combines the program’s object code with other object code to produce an executable binary file

The other object code can come from the Run-Time Library, other pre-compiled libraries, or object files that you have created

On a Unix system, the executable file is called a.out by default If any linking errors are thrown, no executable file will

be generated

Linking

Translation Process

Static Linking

libm.a

printf.o, fopen.o

Linker (ld)

foo.obar.o

libc.a

a.out

fully linked executable object file

Merging Relocatable Object Filesinto an Executable Object File

main()m.o

int *ep = &ea()

a.o

int e = 7

headers

main()

a()

0system code

int *ep = &e

int e = 7

system data

more system code

int x = 15int y

system data

int x = 15

Relocatable Object Files Executable Object File

.text

.text

.data

.text

.data

.text

.data

.bss .symtab.debug

.data

uninitialized data .bss

system code

Dynamic Linking

a.out

Linker

Linker

a.o b.o

libfoo.so (position independent shared object)

bar.o

OS Loader (execve)

Dynamic Linker(ld-linux.so)

partially linked executable:dependency on libfoo.so

fully linked executable in memory

-fPIC

Creating a static library

Indexed archive with *.o files Static library for linking: libsomething.a

Create .o files: gcc -c helper.c Create an archive: ar rlv libsomething.a *.o Generate index: ranlib libsomething.a == ar s Link the library: gcc -L/your/dir -lsomething

Creating a dynamic library

Details differ for each platform gcc -shared -fPIC -o libhelper.so *.o

Linking: the same approach as for static (-llibrary) Also via $LD_LIBRARY_PATH, but:

Today used rarely and sometimes disabled completely Security and compatibility issues Replaced by ldconfig /etc/ld.so.conf

Distribution software management systems provide libraries that the installed software depend on

Mostly dynamic libraries Some static libraries – mostly for development Or just to be safe, provide both

Library distribution

Program development using gcc

Source File pgm.c

Program Object Code File pgm.o

Executable File a.out

Preprocessor

Modified Source Code (in RAM)

Compiler

Linker

Other Object Code Files (if any)

Editor

Unix SW development environments GNU Emacs (1985) Kdevelop, Eclipse, Netbeans Xcode Vim (1991) Geany

Other programming tools

There are tools to ease the single programmer development of modular projects (make)

There are version control systems for multi programmer projects: CVS, SVN, Git Git: a distributed revision control system with an emphasis on speed,

data integrity initially designed and developed by Linus Torvalds (2005) for the Linux kernel development

There are packaging tools to ease installing programs: Tarball, RPM, DEB, ...

GNU make utility

If in a directory full of C files there is a text file with the default name makefile then running make will cause it to be used by the make program.

The power of make is that it examines timestamps.

If a change is made only to part2.c and make runs again, only part2.c is compiled. The linker links the old part1.o, old main.o and the new part2.o to make the final program.

Can be used to solve many other timestamp-based issues.

Makefiles

• When programming large applications:– It is better to modularise– Keep individual source files small– Instead of one large file

• Difficult to edit• Slow to compile

– Tiresome to do manually (repeatedly)– Use a Makefile

Example Makefile

• Consider a program contained in two separate source files:– main.c– sum.c– Both include a header sum.h

• Require the executable file to be named sum• A simple makefile can be used (see the next slide)

Example Makefile

# Makes sumsum: main.o sum.o

gcc –o sum main.o sum.omain.o: main.c sum.h

gcc –c main.csum.o: sum.c sum.h

gcc –c sum.c

Comment

Dependency

Action

Dependency line starts with a target at column 0Action line begins with a TAB followed by a command line

Dependency + Action = Rule

make (1976)

Searches the current directory for GNUmakefile/BSDmakefile, makefile, Makefile Runs the specified (or default) target(s) make [–f makefile] [target]

Declarative: necessary end conditions are described but the order in which the actions are to be taken is not specified Maintains dependency graphs (topological sorting of a DAG)

Based on modification times If node newer than child, remake child

make variables and macros

Environment variables are available as $(PATH) etc. Macros can be composed of shell commands by using the

command substitution operator (backticks) Predefined internal macros aka. automatic variables:

$@ name of current target$? list of dependencies newer than target$< name of dependency file$* base name of current target$% for libraries, the name of member

More on make rules Implicit or suffix rules

CC = gccCFLAGS = -g# The special fake target for suffixes.SUFFIXES: .o .c# .c to .o compilation.c.o: $(CC) $(CFLAGS) -c $<

Pattern rules Target contains exactly one character % for matching file names Prerequisites likewise use % to show how their names relate to the target

# From html to txt%.txt: %.html lynx -dump $< > $@

all: hello cleanclean:rm –f *.o

helper.o: helper.c $(CC) $(CFLAGS) -o $@ $< OBJ = helper.o \ hello.ohello: $(OBJ)$(CC) $(CFLAGS) $(LDFLAGS) –o $@ $(OBJ)

make all

The general routine Installing software from the source code generally

involves the following three steps: Configuring the makefile

Generating the makefiles before the compilation to tailor it to the system on which the executable is to be compiled and run

Compiling the code Installing the executables in the appropriate places

./configure make make install

Revision management

Diffutils Find differences among files (diff and diff3) Update a set of files from generated differences (patch)

CVS Maintains a history of changes for groups of files

Including: who, when, what, and optionally why Newer alternative: subversion (SVN)

git – the Linux way

CVS, Concurrent Versions System

Modify source code locally until it’s ready to be merged with what everyone sees

Multiple people can modify the same file and be ok

Develop multiple versions of software simultaneously And merge these branches later on

Recall past versions of code base Based on time ago or version

See committer's comments on their changes

Client/Server

git

Linux kernel needed SCM after the proprietary system BitKeeper withdraw the free licence

Requirements: distributed workflow, speed, safety, open source A non-existent set of features at that time

Created by Linus Torvalds in 2005 Taking CVS as an example of what not to do

If in doubt, make the exact opposite decision ;)

Now de-facto SCM for the UNIX development

Locally stored full repository (revision history)

Packaging approaches:source vs. binary

A software package: a software that is built from source with a package management system (PMS)

Two fundamentally different approaches for packaging-based software distributions:

Providing source packages containing the vendor sources plus instructions for automated build and installation

Providing binary packages containing the final installation files only

Most PMS support both approaches, although often not equally well

source package

binary package

distribution size package size package dependencies

installation reproducability

installationrun-time stability

installation system alignment

installation time

Dependency management is a very helpful feature of package management systems

Keeps the system in a consistent state and guarantees the applications to run in the expected way

rpm or dpkg commands have limited dependency management features

They can report which library a package relies on, but the library can itself rely on other packages…

Repository-based PMS try to solve the dependency hell problem, however, the hell is still faced by the repository maintainers

Dependencies

Main package distribution formats in Linux

There is no standard package manager in Linux Main package management systems:

Tarball files (.tar.gz / .tar.bz2) The old-fashioned way of distributing software in Linux/Unix Usually available for open source projects on their web pages Compatible with all distros The main package manager in Slackware, Gentoo

RPM (RedHat Package Manager) (.rpm) Has been adopted by many other distributions (Fedora, Mandrake, SUSE)

DEB (Debian Package Manager) (.deb) The most popular Linux package format

Installing from Tarball files Software packages coming in source code archives have to be

compiled before installed Usually come in .tar.gz or .tar.bz2 archives

Preserve file system parameters: timestamps, ownership, permissions etc. Unpack the "tape archive":

tar xvf <package_name>.tar tar xzvf <package_name>.tar.gz tar xjvf <package_name>.tar.bz2

cd <package_name> Configure, compile, install:

./configure make make install

README and INSTALL files are typically included

Originally: backup tools tar

-c: create -x: extract -z: use gzip compression -j: use bzip2 compression -v: verbose -f: use file

gzip compression is used in tarballs .tar.gz, .tgz and .tar.Z gzip file: compress file, renaming file to file.gz gunzip file.gz: uncompress file, renaming file.gz to file

bzip2 compression which is slightly better but requires more CPU is used in tarballs .tar.bz2 and .tbz2 bzip2 file: compress file, renaming file to file.bz2 bunzip2 file.bz2: uncompress file, renaming file.bz2 to file

Apache example (historic)1. gunzip apache_xxx.tar.gz2. tar -xvf apache_xxx.tar3. gunzip php-xxx.tar.gz4. tar -xvf php-xxx.tar5. cd apache_xxx6. ./configure --prefix=/www --enable-module=so7. make8. make install9. cd ../php-xxx

10. Now, configure your PHP. This is where you customize your PHP with various options, like which extensions will be enabled. Do a ./configure --help for a list of available options. In our example we'll do a simple configure with Apache 1 and MySQL support. Your path to apxs may differ from our example.

./configure --with-mysql --with-apxs=/www/bin/apxs

11. make12. make install

If you decide to change your configure options after installation, you only need to repeat the last three steps. You only need to restart apache for the new module to take effect. A recompile of Apache is not needed. Note that unless told otherwise, 'make install' will also install PEAR, various PHP tools such as phpize, install the PHP CLI, and more.

Managing software in RedHat-based distros

Packages are installed using the rpm command-line utility Install a package

rpm -i <package_name>.rpm Update an existing package

rpm -U <package_name>.rpm Remove a package

rpm -e <package_name>

yum –Yellowdog Updater, Modified Repository-based package management utility for RPM Used by most rpm-based distros: RHEL, Fedora, CentOS, etc. Solves dependency and update issues Install package and all its dependencies from repository:

yum -y install <package_name>

Three ways to manage packages in Debian

dpkg: used on .deb files (like rpm) Standard Unix archives that include 2 tar archives: one holds the control

information and another contains the program data Install: dpkg -i <package_name>.deb

If an older version of the package is installed it updates it automatically by replacing it with the new one

Remove: dpkg -r <package_name>

dselect: a front-ent to dpkg, superseded by APT

apt-get: a front-end to dpkg,the most frequent way of managing software packages in Debian

Install: apt-get install <package_name> Remove: apt-get remove <package_name>

Package management: features Using RPM

Install or upgrade a package: rpm -Uvh <package>-1.0.i386.rpm Remove a package: rpm -e <package>-1.0 Determine the version of a package you have installed: rpm -qa | grep <package> Determine the package a file belongs to: rpm -qf <file>

Using APT Update your package lists: apt-get update Install all updated packages: apt-get dist-upgrade Install a package: apt-get install <package> Upgrade a package: apt-get upgrade <package> Remove a package (and packages that depend on it): apt-get remove <package> Search for a package: apt-cache search <name>

Other Packaging Methods

emerge in Gentoo Linux Deals mostly with source files

Fetches packages and compiles them according to compilation parameters given in /etc/make.conf

e.g. emerge kde #Fetches, compiles and installs packages for KDE

YAST in Suse

Aptitude – a "replacement" for apt-get Better dependency managment and state tracking Less broken systems and unsuccessful installs

Gentoo Portage The emerge command-line tool is the key tool Provides ebuild scripts that download, patch, compile, and install

packages

Modeled on the ports-based BSD distributions Used also in Chrome OS

Source code tarballs are downloaded No need to wait for someone to make a binary package for your distribution

Dependency checking, extreme customization

Users specify what they want, and the system is built to their needs Compilers are optimized for the specific hardware

e.g. Altivec on G4 PPC chips, Pentium versus Athlon

Portage tree

A collection of ebuilds, files that contain all information Portage needs to maintain software install, search, query, ... ebuilds reside in /usr/portage/ by default

The Portage tree is usually updated with rsync: $ emerge --sync $ emerge-webrsync

To search for all packages who have "pdf" in their name: $ emerge search pdf

Evaluation of application

Edit RPM Specification

(.spec)

Commit RPM Specification

to CVS

Checkout RPM Specification

from CVS

Fetch Vendor Sources

Roll Source RPM Package

Store Source RPM into

Repository

Fetch Source RPM from Repository

UnpackSource RPM

Unpack Vendor Sources

Configure & Build Application

Install Application

Store BinaryRPM into

Repository

Roll Binary RPM Package

Fetch Binary RPM from Repository

Install Application

into Instance

RemoveApplication

from Instance

upgrade

END

BEGIN

Track Vendor Sources

Package lifecycledeveloper

administrator

On-line package repositories Large package bases on the Web

Accessible via FTP or HTTP

http://rpm.pbone.net http://www.apt-get.org

You can learn how to build Your own distributionhttp://www.linuxfromscratch.org

Using rpm

rpm –i install package, check for dependencies

rpm –e erase package rpm –U upgrade package rpm –q query packages (e.g., -a = all)

rpm -qrpm -q -i telnetName : telnet Relocations: (not relocateable)Version : 0.17 Vendor: Red Hat, Inc.Release : 18.1 Build Date: Wed Aug 15 15:08:03 2001Install date: Fri Feb 8 16:50:03 2002 Build Host: stripples.devel.redhat.comGroup : Applications/Internet Source RPM: telnet-0.17-18.1.src.rpmSize : 88104 License: BSDPackager : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>Summary : The client program for the telnet remote login protocol.Description :Telnet is a popular protocol for logging into remote systems over theInternet. The telnet package provides a command line telnet client.

Install the telnet package if you want to telnet to remote machines.

This version has support for IPv6.

Building your own rpm: spec

# # spec file for hello world app# Summary: hello worldName: helloVersion: 1.0 Release: 1Copyright: GPL Group: Applications/TestSource: http://www.cs.columbia.edu/IRT/software/URL: http://www.cs.columbia.edu/IRT/software/Distribution: Columbia UniversityVendor: IRTPackager: Henning Schulzrinne <hgs@cs.columbia.edu>BuildRoot: /home/hgs/src/rpm

%description The world's most famous C program.

Building your own rpm: spec

%preprm -rf $RPM_BUILD_DIR/hello-1.0zcat $RPM_SOURCE_DIR/hello-1.0.tgz | tar -xvf -

%buildmake

%installmake ROOT="$RPM_BUILD_ROOT" install

%files%doc README/usr/local/bin/hello/usr/local/man/man1/hello.1

%clean

Building your own rpm

create ~/.rpmmacros%_topdir /home/hgs/src/test/rpm

cd /home/hgs/src/test/rpm/SPECS rpm -ba --buildroot /home/hgs/tmp hello-1.0.spec creates binary and source RPM

Most popular package manager in the world APT is a system created in the Debian

community to automatically manage the packages dependencies

APT can install, remove and upgrade packages, managing dependencies and downloading the packages

It’s a frontend to other tools, and it uses the underlying package management system, like the rpm or dpkg commands

It’s able to fetch packages from several media (cdrom, ftp, http, nfs), and it can be used to create ad-hoc software repositories

APT

APT – Using (1/3)

[root]@[/] # apt-get install nautilusReading Package Lists... DoneBuilding Dependency Tree... DoneThe following extra packages will be installed: bonobo libmedusa0 libnautilus0The following NEW packages will be installed: bonobo libmedusa0 libnautilus0 nautilus0 packages upgraded, 4 newly installed, 0 to remove and 1

not upgraded.Need to get 8329kB of archives. After unpacking 17.2MB

will be used.Do you want to continue? [Y/n]

APT – Using (2/3)

[root]@[/] # apt-get remove gnome-panelReading Package Lists... DoneBuilding Dependency Tree... DoneThe following packages will be REMOVED: gnome-applets gnome-panel gnome-panel-data gnome-

session0 packages upgraded, 0 newly installed, 4 to remove and 1

not upgraded.Need to get 0B of archives. After unpacking 14.6MB will be

freed.Do you want to continue? [Y/n]

APT – Using (3/3)

[root]@[/] # apt-cache search pdfkghostview - PostScript viewer for KDEtetex - The TeX text formatting system.xpdf - A PDF file viewer for the X Window System…

[root]@[/] # apt-cache show xpdf…Filename: xpdf-1.00-3.i386.rpmDescription: A PDF file viewer for the X Window System.Xpdf is an X Window System based viewer for Portable

Document Format (PDF) files. Xpdf is a small and efficient program which uses standard X fonts.

FreeBSD – Hybrid package management Binary packages

Uses FreeBSD repositories Common ./configure options for compilation pkg_add -r <package_name> pkg_info pkg_delete <package_name>\* Rather simplistic, no updates

pkg_delete <package_name>pkg_add -r <package_name>

pkgng – new generation pkgsolves lots of issuesnew advanced functionality

FreeBSD – Ports Collection (Source packages)

Download snapshot (index) of ports collection from FreeBSD mirrors portsnap fetch

Extract index as directory tree to /usr/ports portsnap extract

Install software package cd /usr/ports/*/lighttpd make install clean Downloads and installs (compiles) tarballs (also all

dependencies) Asks for configuration options (frontend for ./configure)

FreeBSD – what is Port?

Contents of directory in /usr/ports/<cat>/<name>

Some info files Dependency list Link to download original tarball from developer site

List of tarball files and SHA256 Patch files Configuration files Configuration options Makefile which does all the work