User documentation for designing online and offline ...€¦ · DEL-055, February 1st, 2013 User...

GLORIA is funded by the European Union 7th Framework Programme

(FP7/2007-2013) under grant agreement n° 283783

User documentation for designing online and offline experiments

CODE: DEL-055

VERSION: 01

DATE: February 1st, 2013

DEL-055, February 1st, 2013 User documentation for designing online and offline experiments

Authors: Aleksander Filip ZARNECKI (UNIWARSAW)Alberto J. CASTRO-TIRADO (CSIC)Robert SIMPSON (UOXF)

Collaborators: Chris LINTOTT (UOXF)Lech MANKIEWICZ (UNIWARSAW)Marcin ZAREMBA (UNIWARSAW)Mikolaj CWIOK (UNIWARSAW)Arkadiusz CWIEK (UNIWARSAW)Ariel MAJCHER (UNIWARSAW)

Revised by: Alberto J. CASTRO-TIRADO (CSIC)Robert SIMSON (UOXF)

Approved by: Aleksander Filip ZARNECKI (UNIWARSAW)

Distribution List:

Name Affiliation DateFernando Serena UPM January 19, 2013

Esteban González Guardia UPM January 19, 2013

Alberto J. CASTRO-TIRADO CSIC January 19, 2013

Chris Lintott UOXF January 19, 2013

Juan Cabello UMA January 19, 2013

Lech MANKIEWICZ UNIWARSAW January 19, 2013

Martin Jelinek CSIC January 19, 2013

Michael Prouz IP-ASCR January 19, 2013

Miquel Serra-Ricart IAC January 19, 2013

Robert Simpson UOXF January 19, 2013

Francisco Manuel SANCHEZ MORENO

UPM January 28, 2013

http://gloria-project.eu 3/14


Change Control

Issue Date Section Page Change Description01 01/02/2013 All All First version of the document completed

Reference Documents

Nº Document Name Code VersionR1 D2.1 - Terms and conditions for members and users of the network DEL-014 01

R2 D3.7 - Proposal for user access to network resources rules DEL-029 01

R3 D4.3 - FAQs, HowTos and manuals (for scientific users). DEL-030 01

R4 D9.15 - Peer-to-peer data storage software Specification and source code DEL-050 01

R5 D3.10 - Report for standard methodology for online experimentation and the choice of the on-line experiments

DEL-023 01

R6 D8.4 - On-line demonstrator experiment implemented DEL-040 01

R7 D3.11 - Report for standard methodology for offline experimentation and the choice of the off-line experiments

DEL-024 01

R8 D8.5 - Off-line demonstrator experiment on educational level implemented

DEL-041 01

R9 D8.6 - Off-line demonstrator experiment on research level implemented DEL-049 01

R10 D7.4 - Software User Manual DEL-026 01.A

R11 D7.1 - RTS2 based OS DEL-012 01

R12 D5.3 - GLORIA Deployment and support website DEL-032 01

R13 D8.8 - Documentation of data reduction methods used by RT in the network MAN-01301 01

R14 D8.1 - Web components and documentation for programming on-line experiments

DEL-033 01

R15 D7.6 - Manual for Creating New On-line experiments with a RT network and Ciclope Astro

MAN-01601 01

R16 D8.2- Web components and documentation for programming off-line experiments

DEL-034 01

R17 D8.3 Web components and documentation for preparing community-wide experiments

DEL-039 01

R18 Liferay Manual MAN-012 01



Index1. Introduction...............................................................................................................................................................72. GLORIA user documentation...................................................................................................................................7

2.1. User accounts....................................................................................................................................................72.2. Access to resources...........................................................................................................................................82.3. Troubleshooting................................................................................................................................................9

3. Documentation on GLORIA resources.....................................................................................................................93.1. Telescopes and wide-field systems...................................................................................................................93.2. Data archives....................................................................................................................................................93.3. Demonstrator experiments.............................................................................................................................10

4. Documentation of GLORIA tools...........................................................................................................................124.1. Telescope control............................................................................................................................................124.2. Scheduler........................................................................................................................................................124.3. Analysis framework........................................................................................................................................12

5. Documentation of GLORIA web components.......................................................................................................135.1. On-line experiment web components.............................................................................................................135.2. Off-line experiments web components..........................................................................................................13

Figures IndexFigure 1: The GLORIA users support entry webpage..................................................................................................8

Figure 2: A typical CASANDRA-3 all.sky image taken at the BOOTES-3 astronomical station in Blenheim (New Zealand), wtih the 0.6m Yock-Allen telescope pointing to the Galactic Center..........................................11



1. Introduction

The main goal of the GLORIA project is to build a community of citizen scientists around an intelligent network of robotic telescopes, fully controlled via the Internet. For attracting amateur astronomers to GLORIA it is vital to have all tools and resources properly documented. The documentation is created simultaneously with software tools developed within the network. It is already available to GLORIA users to a large extent, and was also described or included as a contribution to many previous deliverables. The main repository of documentation available to GLORIA users is the GLORIA Support web page available at:

http :// gloria - project . eu / web / gloria - support

The snapshot of the main support page, showing the available contents is presented in Figure 1 (next page).

This document summarizes the available documentation focusing on the needs of a user planning to design new online or offline experiments. Before a new experiment can be implemented on the GLORIA network, user have to make themselves familiar with:

● user account policy and access rights,

● resources available in the network, which can be used to build experiments,

● tools developed within the network to run experiments,

● web components designed to allow users to implement the designed experiments.

The documentation available for these areas of interest is described in subsequent sections.

2. GLORIA user documentation

The GLORIA web site is the main access point for the partners and the users. It is divided into three main sections: a private section for internal use; a public section used for communications, outreach and interaction of the GLORIA partners with the general public; and a GLORIA users' section. Without any restrictions all users will have access to GLORIA documentation in the public section. The User section, is accessible only for registered users and is where they can perform "experiments" allowed by GLORIA: real-time observations, batch observations and off-line images and data browsing and analysis.

2.1. User accounts

User accounts are created by GLORIA administrators based on the user request via a dedicated form. “Personal” or “group” user accounts can be created. In both cases only one person will be identified as responsible for the account, having to accept Terms and Conditions for using GLORIA resources, see Deliverable 2.1 (R1) for detailed description. User activities, in particular those related to the usage of GLORIA resources, will be monitored and logged. The main aim is to monitor user activities in general, to understand their needs and improve network performance. However, we would also like to detect possible anomalous usage, which can downgrade network performance.


http://pre.gloria-project.eu/web/gloria-support














2.2. Access to resources

One of the problems faced by GLORIA is how to distribute resources between users. The most valuable resource is telescope observation time, as the number of robotic telescopes is much lower than the number of potential users and only one user is able to control a telescope at a given time. However, in case of large demand, similar problems can be faced for computing or storage resources. To determine the resources a user receives (e.g. observation time), GLORIA needs to


Figure 1: The GLORIA users support entry webpage.


evaluate users in some way, so that the resources can be assigned according to the priorities based on the evaluation of each. At this point, the concept of 'karma' is introduced.

Karma is an index that measures the reputation of a user in GLORIA. A high karma value means a high reputation. The karma index is calculated automatically by the GLORIA System, based on user activities and the response received from other users (Social Activity component; for details see Deliverable 3.7 (R2)).

The remaining problem, which has to be solved is how to assign observation time (and possible other resources) as a function of the user karma value. One could try to define a complex and large set of rules to do this, which would however not be very efficient nor flexible. A more elegant approximation has been chosen for GLORIA, using the bidding mechanism (auctions). This is thought to result in a most efficient usage of GLORIA resources. Detailed proposal for user access to network resources rules, with detailed description of the bidding rules, can be found in Deliverable 3.7 (R2).

2.3. Troubleshooting

The GLORIA Support web page includes, in addition to extended 'HOW TO' documentation, manuals and other downloadable material and a dedicated Frequently Asked Questions (FAQs) section. The first version of it was presented in Deliverable 4.3 (R3), but we will continue to develop up to the end of the project. The aim is to cover all identified common problems, which are met by GLORIA users. If the user is unable to find a solution to their problem there, they can also post the description of the problem on GLORIA forum or contact directly responsible GLORIA partner.

3. Documentation on GLORIA resources

3.1. Telescopes and wide-field systems

For the best usage of GLORIA resources, in particular telescope observing time, it is vital that users have detailed information of telescope capacities and constraints. The general description of the GLORIA infrastructure is available on the project web page:

http :// gloria - project . eu / about / telescopes -2/

More information can be also found on the web pages of partner observatories, which are referenced there as well.

3.2. Data archives

All observations done by GLORIA users within GLORIA experiments will be stored within GLORIA storage resources - at least for the time needed for a user to complete their analysis. We would like to enable permanent storage of high quality sky images coming from GLORIA telescopes, but can not promise this at present, due to finite resources. In any case, to allow other users to use these data for their studies (offline experiments) we need to maintain well defined data format, quality monitoring, transparent storage structure and access policy.


http://gloria-project.eu/about/telescopes-2/













We also hope that GLORIA users will want to share their own storage space (e.g. related to the their RTS) with the community. The GLORIA System will be able to access the shared space through a set of specific web services. These web services should be completely open regardless of file systems and space distribution. The guide for users willing to share their resources is available on GLORIA Support web page.

One of the main goals of GLORIA is also to develop a dedicated storage solution that provides a media repository where media content is stored persistently and accessible for the GLORIA users. This platform should be scalable and able to handle a considerable number of GLORIA users concurrently, regardless of the overhead and current performance of the network. We decided that the platform should be divided into two separate solutions: the cloud-based solution where a pool of servers manage the media repository and the peer-to-peer solution where GLORIA users can contribute by donating their resources. The two solutions are combined into a single distributed storage platform to fulfil all the requirements for the distributed storage. Detailed description of the proposed solution can be be found in Deliverable 9.15 (R4).

3.3. Demonstrator experiments

Online experiments are how the users access GLORIA's telescopes. There are, in general, six possible types of observations, which can be performed, depending on the choice of two key options: when is the observation performed (at fixed time, according to automatically generated schedule or in response to external alert) and how it is done (interactive observation by user or batch execution of predefined script). Detailed description of all possible modes of operation is included in Deliverable 3.10 (R5).

Two of the GLORIA telescopes are equipped with specialist equipment required in order to image the solar surface - the Solar TAD (http :// www . ot - tad . com ) and Montegancedo Robotic Telescope (http :// om . fi . upm . es ). In order to present functionality provided in the GLORIA network, a demonstration online experiment, based on these two observatories, was implemented. This experiment makes use of the web tools developed within GLORIA to allow users to take Sun images and record the number of individual sunspots and sunspot groups, in order to make an estimate of total solar activity. As with many online experiments, the same tools and methodology can be used to investigate archive data as an offline experiment, and an extension which makes use of an automatic analysis pipeline is also possible. Detailed description of tools used to implement this demonstration online experiment is included in Deliverable 8.4 (R6).

There are several different categories of offline experiment as well, involving different activities on the part of the user and the database, which can be considered either at educational or at research level, depending on the expected outcome. Methodology and possible choices of experiments are described in Deliverable 3.11 (R7). We assume that the image analysis framework which is being developed for GLORIA, will be sufficiently flexible to cover all (or at least most) concepts proposed. Two experiments, one on educational and one on research level, were chosen as offline demonstrators.

Planet orbit reconstruction with all-sky images from the CASANDRA instruments at the BOOTES stations (http://bootes.iaa.es) was selected as the educational level offline demonstrator. A web component will show data from an all-sky camera for a single night at a given time (see Fig. 2 as an


http://bootes.iaa.es/

http://om.fi.upm.es/









http://www.ot-tad.com/










example). The user can draw on the image to mark the selected bright objects and then load a new image, with data from the following night. Planets can be easily recognized as the objects smoothly changing their position between images. The user’s annotations are stored as a track on the image, as they continue to mark the progress of the planets on successive nights. This allows users to explore ideas of planetary motion, including relative speed across the sky, the ecliptic, and retrograde motion in some cases. Details of the implementation of this experiment were described in Deliverable 8.5 (R8).

The experiment dedicated to light curve reconstruction and classification of variable objects was selected as the research level offline demonstrator experiment, to be implemented based on the Pi of the Sky data (http://grb.fuw.edu.pl). The concept of the experiment is described in details in Deliverable 3.11 (R7). As the Pi of the Sky instruments were developed to monitor large regions of the sky in order to detect powerful transients known as gamma-ray bursts, they have some of the largest fields of view of any of the GLORIA instruments, making them suitable for this experiment. A set of about 300 archive images of the pre-selected sky fields was used as an input to the Luiza data analysis package. Luiza includes tools for photometry, astrometry and lightcurve reconstruction. A lightcurve of an object selected by the user can be then displayed using dedicated


Figure 2: A typical CASANDRA-3 all.sky image taken at the BOOTES-3 astronomical station in Blenheim (New Zealand), wtih the 0.6m Yock-Allen telescope pointing to the

Galactic Center.

http://grb.fuw.edu.pl/


browser-based tools. Detailed description of the offline research level demonstrator experiment implementation can be found in Deliverable 8.6 (R9).

4. Documentation of GLORIA tools

A regular user interacts directly with GLORIA System during its user experience through the User Account Management, any available tools and services, Experiments and depending on user role, also the Telescope Management. General description of the tools and software available to users is described in Software User manual (Deliverable 7.4 (R10)). From the point of view of online and offline experiment design, the following services are of primary importance:

4.1. Telescope control

RTS2 (Robotic Telescope System 2) has been chosen as the reference control system for RTs in the GLORIA network. Its structure is documented in detail in Deliverable 7.1 (R11). RTS2 is based on the “plug-and-play” philosophy. The parts which constitute the system can be started, restarted or stopped anytime, without affecting system performance. Four basic type of components include: a central daemon for the system; services to interact with database, execute an observation, process images; devices to drive cameras, mounts, domes, weather sensors, etc; and clients which provide information to end-users, and are usually running in interactive mode. RTS2 devices correspond to the different devices a robotic telescope can have. They are classified into types such us cameras, mounts and domes and have a set of constant and variable parameters that define their characteristics and status. A user designing an online experiment has to identify devices required for making the observation and relations between their functionalities. Also important from the user point of view is the RTI (Remote Telescope Interface) which defines the interface which GLORIA web tools use to interact with the remote telescope. Details of the RTI web services configuration are included in Deliverable 5.3 (R12).

4.2. Scheduler

In Batch mode, a given GLORIA telescope receives an observing script at the beginning of a block of observing time, and executes it via a local interpreter without further interaction with the GLORIA system. The observing script is generated based on Observing Plans submitted by GLORIA users. However, for ensuring efficient usage of network resources (observation time) a dedicated scheduler has to be implemented to properly distribute requested observations between the different telescopes in the network taking into account the type of observations proposed and the technical aspects of the available instruments together with the amount of time available at each GLORIA observatory. Here is where the "Intelligent" aspect of the GLORIA concept is - providing a single scheduler that optimises the use of the network as a whole. The scheduler concept and the details of scripting language are described in Deliverable 3.10 (R5).

4.3. Analysis framework

The principal data-handling problem posed by the GLORIA network is the automatic handling of the large variety of (and potentially size of) data from online and offline experiments and large variety of analysis tasks. We need an analysis framework which would be both very efficient and very flexible. Luiza is a simple modular application framework developed for implementing



reconstruction and analysis code. In Luiza, every computing task is implemented as a processor (module) that analyzes the data stored in an internal data structure and created additional output is also added to that collection. The advantage of such a modular approach is to keep things as simple as possible. A user defines the analysis chain at run time, by specifying list of active processors in an XML steering file. The idea is to develop a large number of processors for GLORIA, doing many different tasks, so user is always able to find a set which matches his/her needs. Detailed description of the analysis framework can be found in Deliverable 8.8 (R13).

Luiza is being developed using the Doxygen package for framework documentation. HTML and LaTeX documentation is created automatically from class header files, based on simple tags used in the comments included in the code. Additional work needed to keep the documentation up to date is minimal, assuming developers do put comments in the code anyway. Documentation for the most recent Luiza release is available on the dedicated web page at:

http :// hep . fuw . edu . pl / u / zarnecki / gloria / luiza / doc / html / index . html

5. Documentation of GLORIA web components

High level software developed within GLORIA can be divided into two distinct parts: GLORIA Services and Presentation. Services, as described in the previous section, are responsible for managing the robotic telescope network and implementing all logic of GLORIA experiments. They act as intermediaries between the telescopes and possible GLORIA user interfaces or any external client systems. On top of that, web components form the presentation layer, which is simply the set of front-ends that connect user with GLORIA Services. We are using Liferay Portal technology (www . liferay . com ) to develop and publish these front-ends on the GLORIA website. Basic functionality has been described in a dedicated Liferay Manual (R18).

5.1. Online experiment web components

Web components for online experiments were documented in Deliverable 8.1 (R14). The following general components are being developed: SIMBAD database search interface, GLORIA database search interface, virtual planetarium that can be centred around any location/time, and an all-sky map. Components dedicated to online experiments include: telescope selector, telescope devices control, image viewer/loader, image display with possibly with ability to mark selected objects (e.g. sunspots and groups on Sun images). Manual for Creating New On-line experiments with GLORIA using these components will be included in Deliverable 7.6 (R15).

5.2. Offline experiments web components

Web components for offline experiments were described in Deliverable 8.2 (R16). Components dedicated to off-line experiments include: reader for FITS images/tables, interface for saving result files, search International Virtual Observatory (IVO) database by coordinates, construct Spectral Energy Distributions (SEDs), light curve upload (FITS, CSV, text format) and display, light curve fitting and statistical analysis. Possible use of additional components for developing community-wide experiments is described in Deliverable 8.3 (R17).


http://www.liferay.com/





http://hep.fuw.edu.pl/u/zarnecki/gloria/luiza/doc/html/index.html


























GLORIA Partners

UPMUniversidad Politécnica de MadridSPAIN

AUAVAstronomical Institute, Academy of Sciences of the Czech RepublicCZECH REPUBLIC

CSICConsejo Superior de Investigaciones CientíficasSPAIN

INAT & CVUTCzech Technical University in PragueCZECH REPUBLIC

IP-ASCRInstitute of Physics of the Academy of Sciences of the Czech RepublicCZECH REPUBLIC

IACInstituto de Astrofísica de CanariasSPAIN

INAFIstituto Nazionale di AstrofisicaITALY

SAOSpecial Astrophysical Observatory of Russian Academy of SciencesRUSSIA

UCDNUIDUniversity College DublinIRELAND

UCUniversity of ChileCHILE

UMAUniversity of MalagaSPAIN

UOXFUniversity of OxfordUNITED KINGDOM

UNIWARSAWUniwersytet Warszawski POLAND


User documentation for designing online and offline ...€¦ · DEL-055, February 1st, 2013 User...

Documents

Transcript of User documentation for designing online and offline ...€¦ · DEL-055, February 1st, 2013 User...