FLOSS Pilot Studies

Software Engineering Group:: http://sweng.csd.auth.gr/

IInvolving Students in the F/OSS Learning nvolving Students in the F/OSS Learning Environment: Environment: Encounters and ExperiencesEncounters and Experiences

FLOSSCOM

Thessaloniki, Greece.

November 13th – November 14th, 2006

Sulayman K. Sowe [email protected]


Introduction

• Software Engineering Education (SE) Now– Inspired by closed-source software development in lecture

formats. – Teaching specific SE skills in semesters.– Project-based coursework. Students work in industry to

gain practical experience.• Challenges

– Expose students to the SE principles and techniques we teach them.

– Give students practice in large and long-term projects.– Will industry scarify their software (code) to students?


• What we want to do– Run a small pilot study in F/OSS mode.– Students as volunteers in software testing in F/OSS projects.– Give students opportunity to work on what they considered

interesting. – Give students ‘real-world’ experience in dealing with F/OSS

projects.• Roles

– Students: find and report bugs.– Lecturer: provide useful guidance and support.


• Implemented in the Introduction to Software Engineering (ISE) course.

• Duration: 12.5 weeks in 5th Semester in 2005.• 15 students volunteered• 1 lecturer + 1 adviser.

F/OSS Teaching & Learning Framework


• Introduction to F/OSS (8hrs)– What is F/OSS?, projects, activities, testing, forums, etc.

• Exploratory process– Students browse projects portals (sourceforge.net)

• Projects selection criteria– Operating system, # of developers, development status (α/β),

active mailing lists/forums. • 1st class presentation

– Brief history of project, bug reporting procedure, testing tools used.

F/OSS Teaching & Learning Framework– Phase 1


F/OSS Teaching & Learning Framework– Phase 2[Students Involvement in Testing]

2nd class presentation (Week 5)– Types of bugs found, how they were found, what caused them,

how they were reported, any responses?


F/OSS Teaching & Learning Framework – Phase 3

• 3rd class presentation (Week 12)– student particulars (Project name, login id, website, history,

screenshots)– List of testing activities:

• bugs found (bfn), bugs reported (brp), bugs fixed (bfx), replies (rep) received, Urls of brp, bfx, and rep.

– Likes, dislikes and future plans (if any) in the project.


F/OSS Teaching & Learning Framework – Phase 3

• Grading: Coursework (50%)

– Presentations = 10% • 3 pts for each presentation in phase 1 & 2. • 4 pts for presentation in phase 3.

– Project participation = 12%• # of emails exchanged with the student about his

project.

– Working with testing tools = 13% • using and understanding bug tracking systems/bug

databases.

– Testing activities = 15%• measured by 3 variables (bfn, brp, and rep)


F/OSS Framework Evaluation - 1Evaluation Approach = Testing activities + Online Surveys

1. Testing activities. Students as bug seekers

43156872Sum

2.1751.2812.7433.017Std. Dev.

104.03.0Mode

3.001.004.004.00Median

3.311.155.235.54Mean

13131313No. of

students

repbfxbrpbfn

• bfn > brp, some bugs found were already reported.

• Not required but some students did well in bfx.

# Project Category Url1 Imagein Image Processing http://sourceforge .net/projects/imagein

2 Mozilla (Firefox) Internet Browser http://wiki.mozilla.org/Firefox:1.5.0.2:Test_Plan

3 Mozilla (Seamonkey) Internet Browser http://www.mozilla.org/projects/seamonkey/4 Vdrift Games http://vdrift.net/

5 Cube Games

6 Eclipse O pen Platform http://www.eclipse .org/

7 FloAts Mobile Agent Mobiles & Networks

8 Torcs Games http://torcs.sourceforge.net/

9 Audacity Entertainment http://audacity.sourceforge.net/

10 Mednafen Games http://mednafen.com/

11 Astronomy https://sourceforge .net/projects/ste llarium

12 Dr.DivX Playback http://sourceforge .net/projects/drdivx/

13 Mill3d Games14 Stunts3D Games http://sourceforge .net/projects/stunts3d

15 Mega Mario Games

16 Gloster Games

http://sourceforge .net/projects/cube

http://fma.sourceforge.net/index2.htm

Stellarium

http://sourceforge .net/projects/mill3d

http://sourceforge .net/projects/mmario

http://gloster.sourceforge.net


F/OSS Framework Evaluation - 2

.353p

.281rbfx

.089.911p

-.035rbrp

.024.730.001p

.620.106.960rbfn

repbfxbrp

• Correlation analysis (Bivariate)

• bfx uncorrelated with bfn, brp, rep.

• bfn vs brp (r=0.690, p=0.001).

• brp vs rep (r=0.490, p=0.089). Interaction with community is moderate.

.490


• Factor Analysis (Principal Component Analysis & Varimax rotation)


• Students’ behaviour explained in terms of 2 latent factors.

• f1: bfn, brp and to some extent rep

• f2: bfxThe factor scores estimated by the Anderson-Rubin method


• Student #2 high score in f1 & f2…good performance

• Student #6 high score in f1 low score in f2

• Students #3, #5, #8, #9, #10, and #12…moderate performance

• Students #4 and #11 high score in f2……bug fixers

F/OSS Framework Evaluation – 4. Factor Analysis Cont.


Online Surveys

• 2 online surveys (Week 6 & 13). N=11 for both surveys

• 7 questions in Survey 1 were repeated in Survey 2

Count

11 11

11 11

22 22

survey 1

survey 2

survey

Total

Yes

Enjoytesting

Total

Count

7 4 11

7 4 11

14 8 22

survey 1

survey 2

survey

Total

No Yes

Easy to get a project

Total

Count

6 4 10

6 5 11

12 9 21

survey 1

survey 2

survey

Total

No Yes

Easy to find bugs

Total

Count

1 10 11

2 9 11

3 19 22

survey 1

survey 2

survey

Total

No Yes

Reporting bugs easy

Total

Count

1 10 11

0 11 11

1 21 22

survey 1

survey 2

survey

Total

No Yes

Prefer other courses inFOSS

Total

Count

0 9 9

2 9 11

2 18 20

survey 1

survey 2

survey

Total

No Yes

Read/understand bugsothers reported

Total

Count

1 8 9

0 11 11

1 19 20

survey 1

survey 2

survey

Total

No Yes

Continue participating

Total

McNemer (NA) (100%)

McNemer (p = 0.549), Not significant

McNemer (p = 0.754), Not Significant

McNemer (p = 0.039), Some significance

McNemer (p = 0.002), Significant

McNemer (p = 0.065), Significant

McNemer (p = 0.008), Not significant



Selected results (Survey 1 & Survey 2)

0 1 2 3 4 5 6 7 8 9 10 11

Able to fix my bugs

Able to fix other bugs

Found bugs after 1-2 days

Discusse bugs w ith lecturer

Lecturer helped in reporting bugs

Helpful Info. on project website

Easy to describe bugs

Got responses from forums

Bug tracker easy to use

Lecturer helped in selecting project

Use BTS to report bugs

Discuss project w ith classmates

Worked w ith other students in project

Easy to get interesting project

Satisfied w ith replies

Enough time to work in project

Satisfied w ith communication mode

Prefer face-to-face communication

Check bug reports every 2 days

Project participants friendly

Communication problems in testing

Item

s

Number of "Yes" Responses

Survey 2

Survey 1

Online Survey Results - 2

Interesting project

Friendly

Response


Experiences & Lessons Learnt - 1

• Advantages• Practical experience

– software testing in the bazaar.– how large and complex F/OSS projects work. – writing good bug reports. – communicate ideas to ‘virtual colleagues’.– working with tools they might use when they graduate.

• Flexible Schedules: test whenever and wherever you want. • Learning Opportunities: learn new tools (e.g. Mantis Bug Tracker).• Sample size: 13, effective interaction with the students.

Face-to-face contact in phase 1 + presentations (brainstorming).



Disadvantages• Specialization: one aspect of the ISE course (software testing). • Collaboration: encouraged but not required to work in

groups/pairs. • Evaluation problem: “Overachievers”. How to reward them? • Sample size: 15 out of 150 volunteered.

• Lecturer as project manager– more than preparing and delivering 2hrs lectures– lecturer as projects manager

Difficulties

• Email/work overload: keeping pace with students emails.

• When to stop scoring points: students “Now stop finding and reporting bugs!” ?



• Communication– IM/Web-based discussion forum.

• Problems: Timing, archiving, searching, etc. • Contradict F/OSS Freedom Norm

– pre-select and present students diversity of projects.• Group work/Community formation

– group students around specified projects .– Identify task-leader based on his contribution.

• Grading/Evaluation– # views a bug report received from the project’s community.

• Problems: Available in few projects. Students view their own submission many times.

Possible Improvements


• Work in progress: – Conduct a full experiment involving larger sample of students

and extend and use the statistical analysis method used in this pilot study.

• F/OSS & SE Education- Projects as Bazaars of learning:● How can we blend the F/OSS teaching and learning

environment with the formal SE teaching and learning context in colleges and universities?

● A broad understanding of the F/OSS pedagogy ● More research of F/OSS evaluation and assessment

methodologies● How F/OSS can improve the quality of teaching and learning. ● How to create a partnership between students and F/OSS

developers, projects and industry?

Conclusion


SELECTED PUBLICATIONS

1. Sowe, S. K., Karoulis, A., Stamelos I. (2005). A constructivist view of knowledge management in open source virtual communities. In Figueiredo, D. A., Paula, A. (eds), Managing Learning in Virtual Settings: The Role of Context. Idea Group, Inc, pp.290-308.

2. Sowe, S. K., Karoulis, A., Stamelos I., Bleris G.L. (2004). Free-Open Source learning community and web-based technologies. IEEE Learning Technology Newsletter, Vol. 6 (1), 2004. pp.26-29.

3. Sowe, S.K., I. Stamelos. (2005). Identification of Knowledge Brokers in OSS Projects Mailing Lists through Social and Collaborative Networks. In Proceedings of 10th Panhellenic Conference in Informatics, Volos, Greece, pp. 132-141.

4. Sowe, S.K., Stamelos, I., Angelis, I. 2006, Identifying knowledge brokers that yield software engineering knowledge in OSS projects, Information and Software Technology, Vol. 48, 2006, pp 1025-1033

5. Sowe, S.K., Stamelos, I., Deligiannis, I. 2006, A Framework for Teaching Software Testing using F/OSS Methodology, in IFIP International Federation for Information Processing, Vol. 203, Open Source Systems, eds. Damiani, E., Fitzerald, B., Scacchi, W., Scott, M., Succi, G., (Boston: Springer), pp. 261-266.

6. Sowe, S.K., I. Stamelos, L. Angelis. 2006, An Empirical Approach to Evaluate Students Participation in Open Source Software Projects. The IADIS CELDA 2006 conference, Barcelona, Spain, 8-10 December 2006.

7. Sowe, S.K., Stamelos . 2006, Involving Software Engineering Students in Free and Open Source Software Projects: Encounters and Experiences. Journal of Information Systems Education, To appear. This work was partially funded through the European Commission, DG Education and Culture, Socrates programme, Minerva action line, project ref: 229405 - CP -1-2006-1- PT - MINERVA – M (http://www.flosscom.net/).

•


Thanks!

Questions?

Comments?

FLOSS Pilot Studies

Economy & Finance

Transcript of FLOSS Pilot Studies