A recommender system for component-based applications using machine learningtechniques

Antonio Jesus Fernandez-Garcıaa, Luis Iribarnea, Antonio Corrala, Javier Criadoa, J. Z. Wangb

aApplied Computing Group, University of Almeria, SpainbCollege of Information Sciences and Technology,

The Pennsylvania State University, USA

Abstract

Software companies are striving more and more to create software that adapts to their users’ requirements. To this end, the devel-opment of component-based interfaces that users’ can compound and customize according to their needs is increasing. However,the market success of these applications is highly dependent on the users’ ability to locate the components useful for them. In ourwork, we propose an approach to address the problem of suggesting the most suitable component for each user at each moment,by creating a recommender system using intelligent data analysis methods. Once we have gathered the interaction data and builta dataset, we address the problem of transforming an original dataset from a real component-based application to an optimizeddataset to apply machine learning algorithms through the application of Feature Engineering techniques and Feature Selectionmethods. Moreover, many aspects, such as contextual information, the use of the application across several devices with manyforms of interaction or the passage of time (components are added or removed over time) are taken into consideration. Once thedataset is optimized, several machine learning algorithms are applied to create recommendation systems. A series of experimentsthat create recommendation models are conducted applying several machine learning algorithms to the optimized dataset (beforeand after applying Feature Selection methods) to determine which recommender model obtains a higher accuracy. Thus, trough thedeployment of the recommendation system that has better results, the likelihood of success of a component-based application inthe market is increased, by allowing users’ to find the most suitable component for them, enhancing their user experience and theapplication engagement.

Keywords: Machine learning, Recommender Systems, Feature engineering, Feature Selection, Component-based interfaces,interaction information acquisition

1. Introduction

Launching a new software application requires an in-depthstudy of many variables in order to successfully achieve a goodposition in the market. Some of these variables are related toaccomplishing a level of design and usability good enough tobecome established in an increasingly competitive market.

Hundreds of projects fail when going to market to the failureof users to embrace them, even when the software applicationcould help them to improve their daily task performance, offer-ing them a service that covers theirs needs. Whether or not com-panies spend a lot of time on design, consume a lot of resourcesenhancing usability or spend large amounts of money on mar-keting campaigns, if users do not quickly find the features theyneed, the software launch is likely to fail. This is more evidenttoday since software applications greatly improved the inter-faces that adapt themselves to the user’s requirements.

The popularity of modern component-based web applica-tions motivates the creation of interfaces that, frequently have

Email addresses: ajfernandez@ual.es (Antonio JesusFernandez-Garcıa), luis.iribarne@ual.es (Luis Iribarne),acorral@ual.es (Antonio Corral), javi.criado@ual.es (Javier Criado),jwang@ist.psu.edu (J. Z. Wang)

a vast amount of components to be discovered. Examples ofthese interfaces are: Google Now [19], a kind of personal as-sistant where each component (called “card”) offer information,answers questions or helps perform actions that users may need;Netflix [29], a streaming video-on-demand platform where eachcomponent offers information on a video’s content [11, 25];Flipboard [14], a news and social media aggregation systempresented in a magazine format; or Fitbit Dashboard [13], anactivity tracker brand that provides a dashboard where users canmonitor their activity, exercise, food, weight and sleep patterns.

Due to the vast number of components that applicationsmay have, the market success of a component-based web ap-plication depends, not only, on the ease with which the usermay find components useful to them but also, on the ability ofthe system to identify the right components at the right time andproperly organize them so they can be at the users’ disposal. Ifusers do not find useful components, they will probably discardthe use of the web application, partly because of their lack ofinterest in the components they know.

The success of a web application can be partially achievedby predicting the components that are most suitable for eachuser at each moment. This can be solved by using simple meth-ods such as identify the most frequently used components. Also,

Preprint submitted to Knowledge-based Systems April 1, 2019

Figure 1: Component-based applications morphology

more elaborate techniques may be applied, for instance, throughcomputational intelligence methods. This is an area yet to beexplored since as far as we know, there are no related works thatexplore the component-based application usage by end-usersthat analyzes behavior to enhance user experience and maxi-mize the software’s probabilities of success in the market.

However, this is not an easy task for several reasons. Withthe passage of time, new components are added to web applica-tions and users should be able to discover these in event that anewly added component may be useful for their interests. Theproblem is more compounded when applications have to makesuggestions to new users without prior knowledge of their ac-tivity. In these scenarios, suggestions should be made based onothers, similar user activities, that could be difficult to identifywithout bias.

The problem is even more challenging today because usersinteract with web applications across several devices (smart-phones, tablets, etc.) using several forms of interactions (touch,gesture, voice, etc.). The devices that users handle and the waythey interact with them clearly affect users’ behavior. Often itis not enough to identify which component may be suitable foreach situation but also whether the usage of that component isappropriate to the form of interaction and the device that a useris handling at that specific time.

In addition, more aspects may be considered to be in pre-dicting the suitability of components to users: a) The natureof the user interacting may be taken into consideration (userscategorization); b) Information may be extracted from the sur-rounding environment (context awareness); and c) other rele-vant external information related to the web application may beidentified that may enrich the prediction data.

This paper addresses the problem of creating a useful rec-ommendation system [31] that would be able to forecast theuse of components in cross-device component-based applica-tions with multiple forms of interactions. We intend to create arecommender system that can help users to discover the com-ponents most suitable for them, thereby improving their userexperience of the applications. The success of the recommen-dation system will likely increase the possibilities of a web ap-plication project being successfully introduced into the market.

To validate the effectiveness of the recommendation systemand the data analysis methods applied to suggest an example setof components to users, we have performed an empirical studyon a real component-based web application with tracked data ofits previous users. The name of the application is ENIA (Envi-ronmental Information Agent) [1], a component-based user in-terface for environmental management used by the AndalusianEnvironmental Information Network (REDIAM, Spain) [3].

In this case study, a practical methodology is applied thatin early stages, obtains a raw dataset from the ENIA appli-cation. To this dataset, Feature Engineering techniques suchas deleting features, filtering instances, transforming featuresdatatypes, and merging or splitting features, among others areapplied. This is fundamental for the proper application of ma-chine learning, because the features representation has a greatimpact on improving prediction models. After that, the MutualInformation Score and Chi-Squared Statistics methods are ap-plied to obtain a subset of the dataset previously transformed bythe Feature Engineering techniques. These Feature Selectionmethods automatically select the most relevant features in thedataset and simultaneously, together with the Feature Engineer-ing techniques, reduce a dimensionality problem that appears inour case study dataset.

The rest of the article is organized as follows. Section 2reviews some related projects that involve creating forecast-ing systems or model-based recommendations models. Sec-tion 3 describes the problem of suggesting components to usersof components-based user interfaces in depth and details themethodology followed summarizing how the solution to theproblem is addressed. Section 4 details everything from gath-ering raw data to the creation of optimal datasets. The maintopics discussed in this section are: a) gathering and describingthe raw dataset obtained from ENIA web application; b) apply-ing feature engineering techniques to have transform featureswith better representation; and c) applying feature selectionsapproaches to identify the most significant features. Section 5describes the algorithm used to build the recommendation mod-els and analyzes the experimental results through relevant met-rics. Finally, some conclusions and further considerations aresummarized in Section 6.

2. Related Work

The evolution of data mining and big data involves an in-creasing interest in forecasting the demand for a product, theneeds of a user or the possibility of a series of events happeningin general. Accurately forecasting, precisely identifying trendsand the discovery of behavior patterns clearly optimizes re-source usage or consumption as well as generating new knowl-edge in science and research facilities; enabling faster and bet-ter decisions in politics, retail, weather, sport, science, research,real estate, sports or health-care among many others fields.

For these reasons, the use of recommendation models [31]to forecast the use of resources, anticipate user needs, or tomake recommendations based on their behavior (and that ofother users, including a situation context) are becoming morefrequent. A compilation of some forecasting and recommender

2

models using supervised machine learning algorithms found inthe literature are featured below.

In some works, such as [41], the authors presented a frame-work to improve user recommendation in social networks. Theycapture user preferences involving both interest and social fac-tors to create a model using Matrix Factorization[5] techniques,based on the principle that users who have shown similar inter-ests in the past will likely have similar interests in the future.

Given the difficulty of modeling a real-world recommenda-tion system, sometimes it is useful to create hybrid models thatcombine a model and knowledge of the problem addressed. In[42] the authors created a hybrid model-based job recommen-dation system using Statistical Relational Learning [17]. Theyalso took into account tuning the algorithm to satisfy the re-quirement to give more weight to the more desirable classes (ordifficult ones).

There are also approaches that work with similar algorit-hms used in this article such as Decision Trees [32], LogisticRegression [27] or Artificial Neural Networks [34]. In [7] theyuse data analysis approaches for predicting the sales of newlypublished books and in [24] they use contextual information asa way of recognizing human activities and, subsequently makemusic streaming recommendation accordingly. There are moreworks that focus on multiple purposes such as [44] in which theauthors address the problem of detecting cell mitosis using deepneural networks, or [8] that deal with the detection of maliciouswebmail attachments or [15] that focus on predicting intervalsfor solar energy forecasting with neural networks.

In our study, we also make use of context-aware informa-tion as we consider it to be valuable data. Works that deal withcontext-aware information to create recommendation modelscan also be found in the literature. In [39] the authors pro-pose that contextual features may be considered to be organizedin an understandable and intuitive hierarchy and they exploitthis hierarchical structure to improve the quality of their rec-ommendation models. The exploitation of context informationto improve models is frequently used [38]. For instance, in [35]the authors incorporate contextual situations (e.g., geographi-cal location, special date or activities of interest) to improvethe precision of a retailing recommendation system using a col-laborative filtering approach. In [20] the authors make use ofsmartphone’s context-aware capabilities to create an adaptiveand personalized mobile learning system, which aims to sup-port the semi-automatic adaptation of learning activities andhelps students to successfully complete the learning activitiesof an educational scenario.

Often, before the creation of a recommendation model, Fea-ture Selection techniques are used to improve the dataset qual-ity (among other purposes). We also apply feature selectiontechniques to improve our model’s accuracy, deal with high-dimensional spaces and to solve problems such as the sparsityof data. The literature shows how many works use feature se-lection techniques before building models. There are exam-ples worthy of comment in different areas such as spam detec-tion [43], fraud detection [21] or diagnosis of mechanical faults[40], among others.

Thus, given the related works mentioned, it makes sense

to use machine learning algorithms to create a recommenda-tion model to suggest the most suitable component for users oncomponent-based interfaces, that enhances user experience.

3. Methodology and Problem Description

The steps followed to create the recommendation models oncomponent-based web applications are graphically described inFigure 2. These steps do not represent a formal methodology,they are a guideline we have followed in carrying out the workdescribed in this paper. The application of these steps is com-pletely optional, subject to the nature of the problem presented,and the data scientists knowledge.

The first step, understanding the problem, is already brieflydiscussed in the introduction. It can be summarized as follows:we need to create a recommendation model that assists usersin component-based web applications to discover useful com-ponents, optimizing the user engagement with the application.The aim of the recommendation model is to maximize the po-tential market success of a component-based web application.

In order to really understand the problem of the ENIA com-ponent-based application case study, there are important facts tobe considered (which can be extrapolated to other component-based applications):

(a) The set of components offered by the web application arein continuous growth, new components can be added (orremoved) over time.

(b) New users are registered in the web application with adegree of frequency.

(c) Users are categorized according to how they are expectedto use the application, e.g. Tourist or Farmer.

(d) Users interact with the interface from different devices(Smartphones, Tablets, Wearables, Laptops, Desktops PC,etc.) with different forms of interaction (mouse and key-board, touch screen, voice, etc.).

(e) The context that surrounds the users’ interactions withthe web application is relevant to the way they behave,e.g. Weather.

The second step is gathering the data from the component-based web application [12]. Fortunately, an increasing numberof applications have data repositories and a growing number oforganizations and companies now store data, sometimes leavingit freely available to everyone (OpenData). If this were not thecase, this step would be tedious since it is necessary to accessthe information system and manually collect the data. In thiscase, the ENIA application has a service that stores all of theinformation related to each interaction performed over its userinterface in a MySQL database [30]. A CSV file containing allthe interaction can be requested and will be generated using thestored data in the ENIA database.

The third step consists of processing the raw data obtainedin the previous step. Finding out the relevant attributes that

3

Figure 2: Methodology applied to create the recommendation model

describe an interaction is vital to ensure the success of the therecommendation system. These attributes are called featuresand the set of features is called a dataset. This step includescleaning features and transformation, as well as an importantand laborious process known as Feature Engineering that wewill address further in this paper.

The fourth step consists of applying Feature Selection meth-ods that, as we shall see later in this paper, handle many issuesrelated to data such as the sparsity of dat, in addition to reduc-ing the computing resources needed by the machine learningalgorithms in order to create the models.

The fifth step consists of applying machine learning algori-thms to model the recommendation system we are looking for.There are a wide range of algorithms that can be applied. Inthis work, we make use of decision trees, logistic regressionand artificial neural networks.

Finally, the sixth step consists of validating the recommen-dation model built by analyzing the accuracy of the algorithmsused. For this, a comparison of the experimental results of theapplied algorithms is made to determine whether the recom-mendation models created are suitable to be deployed in thecomponent-based web application and which recommendationmodel should be deployed.

The following sections describe the steps commented above.Section 4 covers the step required to create the datasets thatinclude gathering and processing, apply Feature Engineeringtechniques and Feature Selection methods. Section 5 describesthe algorithms applied to create the recommendation model andthoroughly explains the results and validity of the experiments.

4. Creating Optimal Datasets

In this section, we focus on creating the optimal datasetsfor applying machine learning algorithms. In this way, we shallmaximize the possibilities of suggesting the proper componentto users in every moment. Thus, the recommendation systemthat will be built in further steps will be truly useful to end users.

First of all, we obtain raw data from the ENIA component-based user interface. Following this, we use of Feature En-gineering techniques to help machine learning algorithms to

create accurate and simple prediction models, providing bet-ter results [23]. Finally, we apply Feature Selection methods tocreate a subset with the more meaningful features of the pro-cessed dataset which will imply shorter training times to buildthe models by removing redundant or irrelevant features.

4.1. Gathering Raw DataIn order to create a recommendation system on component-

based applications and suggest useful components to users, itis necessary to create datasets with rich enough attributes tomodel the problem in the first place. As a starting point, wehave a raw dataset provided by our case study component-basedapplication, ENIA. The data contained in the dataset has beendirectly extracted from the component-based web applicationthrough a data acquisition system that is incorporated in theproprietary software application [9, 10] powered by COSCore,a modular and scalable service infrastructure approach for themanagement of component-based architectures [36, 37].

When the dataset was obtained, it consisted of 27702 inter-actions performed (instances) with a set of 18 features. The setof features describing each interaction can be seen in Table 1.

4.2. Feature EngineeringTable 1 describes the data obtained from the component-

based web application. It contains data too raw for direct ap-plication of data analysis techniques. In agreement with theREDIAM agency and their experts in the field, some transfor-mations on the dataset and its features have been made usingFeature Engineering techniques.

Feature Engineering is a process that transforms raw data tocreate features that have better representation and therefore bet-ter to create predictive models [33]. The quality of the predic-tion models created with machine learning algorithms dependson the Feature Engineering approach that has been followed,where usually better features mean better prediction models.

Although Feature Engineering is usually treated lightly, itplays an important role in the success of a machine learningexperiment. These techniques optimize the performance of thedataset, improve the data analysis algorithms’s results, and cre-ate simple and reliable prediction models that perform well andaccurately [23].

4

Feature DescriptionidInteraction Unique Interaction IdentificationdateTime Exact moment in which the interaction took placeoperationPerformed Kind of interaction performed in the applicationlatitude Latitude in which the interaction took placelongitude Longitude in which the interaction took placeidUserClient Unique User IdentificationName Name of the user that performs the interactionSurname Surname of the user that performs the interactionuserType Type of user that performs the interaction (tourist, farmer, etc.)userSubType Subtype of user that performs the interaction (tourist, farmer, etc.)birthDate Birth date of the user that performs the interactioncountryOrigin Country of origin of the user that performs the interactionareaOrigin Area/State of the Country of the user that performs the interactionemail Email of the user that performs the interactiondeviceType Type of device on which the interaction was performed (laptop, desktop, smartphone, etc.)interactionType Form of interaction on which the interaction was performed (mouse&keyboard, touch, voice, etc.)idSession Unique Session IdentifieractionComponent Component over which the interaction has been performed

Table 1: Set of features describing an interaction directly obtained from the ENIA web application

The following Feature Engineering transformations havebeen applied to the original dataset:

(a) No transformation. There are features that have goodsignificance in the form they appear in the original dataset.For this kind of features, no type of transformation is nec-essary. For this reason, the actionComponent, device-Type, interactionForm, countryOrigin, areaOri-gin and userType features have been left unaltered, thevalue they add to the dataset is significant enough as is.

(b) Deleting Features. There are features that do not corre-late well with the objective field (label). Some are easy tofind because they have no knowledge associated with thefield. Others are more difficult to find because they aredependent on other features and to identify them domain-specific knowledge is required. It is important to iden-tify these features and delete them. The idSession,idUserClient and idInteraction features have beendiscarded because they are not relevant to the accuracyof the model, they are simply auto-generated identifierscreated by the ENIA database. The idUserSubType fea-ture has also been discarded because it is an optional fieldthat several users miss. According to the REDIAN ex-perts, ENIA do not provide a good categorization of userssubtypes. The areaOrigin and areaInteraction fea-tures have also been deleted. On this occasion, becausethey have many possible cases, i.e., a large amount ofvalues can be assigned to these features (17 and 12 re-spectively) and we already have the countryOrigin andcountryInteraction features that are similar and havemore homogeneous values. In the future, with a highernumber of instances, they could be kept.

(c) Filtering Instances. Often, the whole set of dataset in-stances are not required because the problem is circum-scribed to a fraction of it. On these occasions, it is neces-sary to obtain a subset of the original dataset according toa rule that can isolate the required instances that are use-ful to create the model. In our case study dataset, the

operationPerformed feature has 13 possible values:{Add, Group, AddGroup, Ungroup, Delete, Ungroup-Group, UngroupDelete, Resize-Bigger, ResizeSma-ller, ResizeShape, Move, Maximize, Minimize}. The-se are the possible operations that users can perform inthe ENIA component-based application user interface.Since our purpose is to recommend components to users,we are interested in filtering the operationPerformed

feature to keep the instances where the operationPer-formed values are {Add, AddGroup}. These values of theoperationPerformed feature bring together the opera-tions where users begin using a component.

(d) Processing Features. There are features that, accordingto domain-specific experts, can be useful but algorithmsmay find them difficult to manage. These features shouldbe processed to adapt them to more suitable datatypes, orrepresentations that increase their significance and maybe easily processed using data analysis algorithms. Inour case study, the birthDate feature is too complex toobtain significance from it, thus it has been transformedto a new feature: the age the user was when he/she per-formed the interaction (age), which performs better.

(e) Creating new features by splitting existing ones. Some-times there are features rich enough to be broken downinto more than one feature because they contain a greatdeal of information in a specific domain. The ENIA orig-inal dataset comes with the dateTime feature, which con-tains the exact time when an interaction was performed.It has been divided into two features: season and part-

OfTheDay, which allows us to explore two different as-pects that surround the interaction and may influence theuser’s behavior.

(f) Creating new features by merging existing ones. Incontrast to the previous case, sometimes there are fea-tures that by themselves do not contribute to creating bet-ter prediction models. However, by merging them with

5

others features, it is possible to create meaningful fea-tures that improve the overall performance. The latitudeand longitude features do not add relevant informa-tion to the recommendation systems by themselves, butthey can be transformed into two interesting new fea-tures: countryInteraction and areaInteraction.

(g) Adding relevant features. Often, there is meaningfuldata that is not contained in the dataset but can be ob-tained from existing features. Because we are talkingabout a component-based web application developed foran Environmental Agency, it is considered relevant, ac-cording to the experts, to know the weather conditions atthe time that the interactions have been carried out. Apiece of code has been implemented that connects to theOpenWeatherMap [2] service and using the latitude,longitude and dateTime data features as inputs can re-trieve the weather conditions. So, two new features havebeen inserted in the dataset: weatherDescription andTemperature.

(h) Discretizing Features. Many machine learning algorit-hms produce better prediction models using discretizedvalues. By discretizing, the impact that small variationshave on the data is reduced, which justifies the loss of in-formation [16]. We performed the following discretiza-tions, assisted by the experts:

— The numeric values of the age feature are calcu-lated (as described before) and categorized into thefollowing values:

{<18, 18-25, 25-35, 35-50, 50-65, >65}.

— The values of the season feature have been catego-rized according to the values:

{Fall, Summer, Spring, Winter}

— The values of the partOfTheDay feature have beencategorized according to the values:

{Morning, Afternoon, Evening, Night}

Table 2 synthesizes the transformation that has been takenplace in each feature.

4.3. Reducing the Number of ClassesClassification algorithms in machine learning can be cat-

egorized as binary or multiclass, according to the number ofclasses they can predict. In this study, we are trying to fore-cast the most feasible components to suggest to users in spe-cific situations. Thus, the algorithm must decide between theavailable components which one is the most suitable. This is amulticlass problem. It would be different if the model were toforesee whether a component is suitable or not, then it wouldbe a much easier binary problem.

After the Feature Engineering process, our dataset consistsof 27702 instances with a set of 11 features (including the ob-jective field). The possible values that can be assigned to eachfeature and its data type is described in Table 3 and summarizedas follow:

Dataset = {

weatherDescription (4),

temperature (7),

countryInteraction (4),

countryOrigin (4),

userType (4),

age (6),

season (4),

partOfTheDay (4),

deviceType (3),

interactionType (3),

actionComponent (33)

}

The number of cases in the actionComponent feature, thatis the objective field, is too high. With 33 different componentsthat can be suggested and 27702 instances to train the model,it is highly probable that the forecasting would not be accurate.According to the number of cases of each feature, the total num-ber of possible cases is more than 50 million (4∗7∗4∗4∗4∗6∗4∗4∗3∗3∗33 = 51.093.504 ≈ 5∗109). If we had not deleted theareaOrigin and areaInteraction features the total numberof possible cases would have been more than 10 american bil-lion (4∗7∗4∗12∗4∗17∗4∗6∗4∗4∗3∗3∗33 = 10.423.074.816 ≈109). With this action the problem was drastically reduced, butnot enough.

For this reason, the number of components to be suggestedhas been limited. A reduction of the 33 possible classes of theobjective field has been made, reducing the number of classes to8. We have selected the components that are the most frequentlyused. The recommendation system is then set up to suggest onlythese 8 components to users. It is remarkable that the appear-ances of these 8 cases selected account for 21882 instances (≈78%). The possible cases of the actionComponents featurethat can be predicted are BioReserves, Clock2, CuttleRoads,GeoParks, Heritage, Twitter, Weather and Wetlands. Thenumber of instances of each class and its frequency can beseen in Table 4. Thus, the recommendation system is a hy-brid between computational intelligence methods and a simplermethod that selects the 8 components most frequently used byusers of ENIA.

Even after the objective field classes reduction, the numberof possible cases is more than 12 million (4∗7∗4∗4∗4∗6∗4∗4 ∗ 3 ∗ 3 ∗ 8 = 12.386.304 ≈ 12 ∗ 106). It remains a large num-ber but, given the high number of occurrences of these classes,and the high appearances frequency of the selected classes, itis enough. The domain-specific experts, i.e., employees of theenvironmental agency that make use of the application to per-form their tasks, agree with this simplification. Besides, fol-lowing their advice, the recommendation model will not sug-gest components to users in every situation, it will only suggeston the occasions that the likelihood of using a component issufficiently high to make a good recommendation. For exam-ple, if the recommender system suggests a component with alikelihood of 37%, the recommendation is not shown to usersbecause we have set a threshold of 50% of likelihood.

6

Before FE After FE ConsiderationidInteraction idInteraction DeletedidUserClient idUser DeletedidSession idSession DeletedoperationPerformed operationPerformed Deleted. Instances filtered by add or addGroupname name Deletedsurname surname Deletedemail email DeleteduserType userType KeptuserSubType userSubType DeletedbirthDate Age ProcesseddeviceType deviceType KeptinteractionType interactionType KeptdateTime season New Feature (Splitting)

partOfTheDay New Feature (Splitting)temperature New Feature (from external sources)weatherDescription New Feature (from external sources)

latitude Country Interaction New Feature (Merging)longitudecountryOrigin countryOrigin KeptareaOrigin areaOrigin DeletedactionComponent actionComponent Objective field

Table 2: Features transformation carried out in the Feature Engineering process

Feature Cases (Possible Values) TypeweatherDescription Clouds, Clear, Fog, Mist Categorical Featuretemperature <-10, -10-0, 0-10, 10-20, 20-30, 30-40, >40 Categorical FeaturecountryInteraction Not limited possibilities. When creating the dataset: 4 cases Categorical FeaturecountryOrigin Not limited possibilities. When creating the dataset: 4 cases Categorical FeatureuserType Tourist, Farmer, Technical, Politician Categorical Featureage <18, 18-25, 26-35, 36-50, 51-65, >65 Categorical Featureseason Spring, Summer, Fall, Winter Categorical FeaturepartOfTheDay Morning, Afternoon, Evening, Night Categorical FeaturedeviceType Computer, SmartPhone, Tablet Categorical FeatureinteractionType MouseKeyboard, Touch, Voice Categorical FeatureactionComponent Not limited possibilities. When creating the dataset: 33 cases. Limited to 8. Categorical Label

Table 3: Set of features describing an Interaction after processing the original raw dataset

Class Instances FrequencyGeoParks 5572 0.25463Heritage 4760 0.21753CuttleRoads 3038 0.13883Twitter 2618 0.11964Weather 2282 0.10428Wetlands 1400 0.06397BioReserves 1302 0.05950Clock2 910 0.04158

Total 21882 1

Table 4: Most frequently used components

4.4. Feature SelectionThe set of attributes of a dataset should be as significant as

possible to describe the nature of the reality they represent. Itmay seem that a large number of features can better describe aproblem and (with them) better predictive models can be built,but this is not entirely true. Feature Selection methods can helpto reduce overfitting [4] and avoid high-dimensional spaces andthe sparsity of data [22] that datasets with a large set of at-tributes may have. Also, a feature subset of a dataset impliesshorter training times building the models by removing redun-dant or irrelevant features.

In this work, we make use of feature selection methods thatsupport all data type features, specifically the Mutual Informa-tion Score and Chi-Squared Statistic methods available in theMicrosoft Azure Machine Learning Studio (AzureML) [28].

4.4.1. Mutual Information ScoreThe Mutual Information Score method is defined as:

I(X; Y) =∑y∈Y

∑x∈X

p(x, y) log(

p(x, y)p(x) p(y)

); (1)

where p(x, y) is the joint probability function of X and Y ,and p(x) and p(y) are the marginal probability distribution func-tions of X and Y respectively [18].

Table 5 shows the results of applying the Mutual Informa-tion Score selection method to the ENIA dataset and Figure 3illustrates the relevance of each feature.

Feature MR MO CR COuserType 0.078864 1 383.256110 1age 0.036541 2 170.996097 2countryInteraction 0.022832 3 106.796770 4weatherDescription 0.020199 4 91.611703 5partOfTheDay 0.018090 5 73.870284 6countryOrigin 0.017928 6 121.178845 3deviceType 0.003489 7 14.995899 7interactionType 0.002720 8 11.730771 8season 0.001766 9 7.233112 9temperature 0 10 0 10

Table 5: Mutual Information Score and Chi-Squared Statistics results (MR:Mutual Information Result; MO: Mutual Information Order; CR: Chi-SquaredResult; and CO: Chi-Squared Order)

7

Figure 3: Mutual Information Score and Chi-Squared Statistic Features Relevance

The relevance of the userType feature is far higher thanany other feature, followed by the age, countryInteractionand weatherDescription features, that also have consider-able weight. Conversely, the deviceType and the interac-

tionType features have little relevance, probably because thevast majority of users interactions has been performed in desk-top computers by means of mouse and keyboard. The tempera-ture feature has no relevance, probably because that featurecan be highly correlated with the season and partOfTheDay

features.

4.4.2. Chi-Squared StatisticThe Chi-Squared Statistic method is defined as:

X2 =

n∑i=1

m∑j=1

(O(i, j) − E(i, j)

)2

E(i, j); (2)

where O(i, j) is the observed value of two nominal variablesand E(i, j) is the expected value of two nominal values. The ex-pected value can be calculated with the following formula:

E(i, j) =

∑ci=1 O(i, j)

∑ck=1 O(k, j)

N; (3)

where∑c

i=1 O(i, j) is the sum of the ith column and∑c

k=1 O(i, j)is the sum of the kth column [26].

Table 5 shows the results of applying the Chi-Squared Statis-tics selection method to the ENIA dataset and Figure 3 illus-trates the relevance of each feature.

When we observe Figure 3 and Table 3 we can appreci-ate that there is not very much difference. Both Mutual Infor-mation Score and Chi-Squared Statistic methods have thrownsimilar results. The main difference is that the weight of thecountryOrigin feature in the Chi-Squared Statistic method ishigher than the countryInteraction and weatherDescrip-tion features in comparison with the Mutual Information Scoremethod.

4.4.3. Features SelectedThe features selected to create a subset of the original dataset

are:

Dataset = {

userType,

age,

countryOrigin,

countryInteraction,

weatherDescription,

partOfTheDay,

deviceType,

interactionType,

actionComponent (label)

}

Taking into account the finding of the Feature Selectionmethods, the deviceType and interactionType features wo-uld have been discarded, but we have kept them since we ex-pect that soon enough, ENIA web application user access fromsmartphones or tablets will increase and we want this recom-mendation system to automatically evolve over time. When cre-ating the models, each is built using both datasets, the originaland the subset of the original with the most significant features.The results are evaluated and their performance compared.

5. Recommendation Model and Experiments

This section analyses the machine learning algorithms usedto create the recommendation models and presents the resultsof the experiments conducted. We evaluate the recommenda-tion model’s accuracy and performance based on relevant met-rics obtained from the empirical study, to determine which rec-ommendation model is more suitable to deploy. By selectingthe more accurate model the recommendation system will pro-vide ENIA users with the most suitable components, therebyimproving their user experience.

8

5.1. Recommendation ModelThere are plenty of algorithms available in the literature that

can be applied in order to create recommendation models. Like-wise, by adjusting the parameters of these algorithms, there aremany more configurations possible that directly affect their per-formance and accuracy.

In this case, it is worth noting that we are facing a multi-class classification problem, where each instance can be classi-fied into more than two classes, specifically, in as many classesas components are available to suggest to users. Although thepossibility exists of turning binary classification algorithms intomulticlass classifiers, we’ve decided to make use of algorithmsthat naturally allow decisions between more than two classes.

We make use of four multiclass classification algorithmsavailable in Microsoft Azure Machine Learning Studio [28].We make use of the AzureML implementation because of theease of setting up web services over trained models to deploythem that this platforms offers. We also value positively the factthat readers can easily access these algorithms as anonymousguests on the platform for free and reproduce the experimentsconducted in this paper.

The well known classification algorithms proposed are Mul-ticlass Decision Forest, Multiclass Decision Jungle, MulticlassNeural Network and Multiclass Logistic Regression.

5.1.1. Multiclass Decision Forest (DF)Decision trees algorithms build a tree-like structure where

each node represents a question over an attribute. The answersto that question create new branches to expand the structureuntil the end of the tree is reached, being the leaf node the onethat indicates the predicted class. The Decision Forest (DF)algorithm, graphically illustrated in Figure 4, creates severaldecision trees and votes the most popular output of them. Theimplementation used in this paper does not directly count theoutput of them but sum the normalized frequency ( f ) of eachoutput in each tree to get the label with more “probability”:

f =1T

T∑t=1

ft(x); (4)

where T is the number of trees and x the probability of eachclass.

The parametrization followed in this experiment built 8 de-cision trees with a maximum depth of 32 levels each. Theamount of 128 splits are generated per node to select the op-timal split. In order to generate a leaf node, just a sample isrequired. To resample the dataset for each decision tree theBagging method, also known as bootstrap aggregation is ap-plied [6].

The bagging method consists of duplicating the original data-set D to a new dataset D′ for each decision tree. D and D′ havethe same size n. When creating and evaluating models the orig-inal dataset is divided in two parts: training and evaluation.The instances used for training and for evaluation are differentin each new dataset D′ since they are selected randomly withreplacement.

Figure 4: Decision Forest Parametrization

5.1.2. Multiclass Decision Jungle (DJ)The Decision Jungle (DJ) algorithm is an extension of the

Decision Forest algorithm where each tree is replaced by a DAG(directed acyclic graph). The structure of a DAG is illustratedin Figure 5. It is more memory-efficient because it eliminatesthe need for repeating leaf nodes and allows branches to mergebut, as a disadvantage, takes more computing time.

Figure 5: DAG Parametrization

The parametrization followed in this experiment built 8 de-cision DAGs with a maximum depth of 32 levels each. Themaximum width of each decision DAG is 128 and the numberof steps for each level optimization of the graph is 2048. As inthe case of the decision forest algorithm, to resample the datasetfor each decision DAG the Bagging method is applied.

5.1.3. Multiclass Artificial Neural Network (ANN)The Artificial Neural Network (ANN) algorithm creates a

set of interconnected levels, where each level consists of a setof nodes (neurons) that receives input and produces weightedoutputs. The nodes of layer 1 are the inputs, the nodes of thelast layer are the output and the nodes in between are called“hidden nodes”. A neural network can be seen as a weighteddirected acyclic graph.

The parametrization followed in this experiment built anANN with a fully connected structure -i.e., every node of each

9

layer is connected with every node of the previous and nextlayer. There is only 1 hidden layer with 100 nodes. The in-stances are processed a maximum of 100 times if the ANN hasnot converged yet. The learning rate is set to 0.1 and the mo-mentum (weight applied to nodes from previous iterations) isset to 0, with these values we want the model to converge fastbut avoid local minimums.

5.1.4. Multiclass Logistic Regression (LR)The Multiclass Logistic Regression (LR) algorithm or Multi-

nomial Logistic Regression), generalizes logistic regression withmore than two possible classes to predict. It predicts the outputprobability of a class by fitting data to a logistic function. LRaims to build a function that describes the relationships betweenthe input features and the class label.

The parametrization followed in this experiment to buildthe logistic regression model set the Optimization Tolerance1 ∗ 10(−6). This parameter set the threshold that each iterationhas to surpass to continue fitting the model to the dataset. TheL1 and L2 regularization weights that deal with the sparsity ofdata and with no sparse data respectively, are set to 1.

5.2. Experiment Results

In this subsection, we present the results of the experimentsconducted that aim to build a useful recommendation model tosuggest the most feasible components to users of a component-based interface software application, thereby enhancing the userexperience and improving their engagement with the interface.Due to the fact that the number of classes to predict was toohigh according to the number of instances of our dataset, wereduced the number of components to suggest from 33 to 8.Thus, the recommendation model will suggest only the 8 mostused components to users in the ENIA application, which are:BioReserves, Clock2, CuttleRoads, GeoParks, Heri-

tage, Twitter, Weather and Wetlands.To build the model, we have selected the four classification

algorithms previously discussed, which are Decision Trees, De-cision Jungle, Artificial Neural Networks and Logistic Regres-sion. We have applied each of these algorithms to the wholedataset after applying the Feature Engineering techniques dis-cussed in section 4.2. We have also applied these algorithmsto a subset of the dataset we have extracted using the MutualInformation Score and Chi-Squared Statistic Feature Selectionmethods described in Section 4.4.

Figure 6 shows the results of the experiments by plotting theaccuracy of the models created with each algorithm, with andwithout reducing the original dataset applying Features Selec-tion methods. The figure clearly indicates that the model builtwith Artificial Neural Networks gets higher accuracy than themodel built using the other algorithms, while the Decision For-est and Decision Jungle tree-like algorithms present the worstresults, especially the Decision Jungle.

Figure 6 also shows that the results are better when applyingFeature Selection methods to the dataset than using the wholedataset. The exact numerical results can be seen in Table 6.We could foresee that a features subset selection of the original

dataset could get better results, given the number of instancesand the number of possible hypothesis. A high sparsity of datain high-dimensional spaces does not only computationally pe-nalizes the creation of a model but also affects to its efficiency,decreasing the accuracy of the model.

Algorithm FS OA AADecision Forest 0.604878049 0.901220Decision Forest X 0.646341463 0.911585Decision Jungle 0.398747390 0.849687Decision Jungle X 0.434237996 0.858559Logistic Regression 0.718978102 0.929745Logistic Regression X 0.740875912 0.935219Neural Network 0.788321168 0.947080Neural Network X 0.802919708 0.950730

Table 6: Overall and Average Accuracy (FS: Feature Selection; OA: OverallAccuracy; AA: Average Accuracy)

DF DJ ANN LR

0.4

0.5

0.6

0.7

0.8

Without FS With FS

Figure 6: Overall Accuracy

Nevertheless, the sparsity of data itself, is not necessarily aproblem if there are insights or knowledge to infer from data,as happens in this scenario. In ENIA, users’ behavior clearlyfollows some patterns, as is borne out by the high accuracyof the models created. Inferring that patterns would be diffi-cult in high-dimensional spaces with the lack of data that ourdataset may have but, by reducing the number of features ofthe dataset with a subset of the most relevance features, we canavoid the problems derived from the curse of dimensionality inhigh-dimensional spaces. Thus, creating models using a subsetof features selected by Feature Selection methods we can getgreater accuracy. The models we have created obtain good re-sults, reaching 80% accuracy in the case of the Artificial NeuralNetwork model. This happens for the models built using everyalgorithm, obtaining better results in all cases with a subset ofthe features of the original dataset.

The good results that every model offers, independently ofthe algorithm used to build it, is remarkable. Even the Deci-sion Jungle algorithm applied to the whole dataset, which is

10

the model that presents least accuracy, throws an accuracy of0.398747390. If one of the 8 possible classes were to be cho-sen randomly, the probability of guessing would be 0.125. Itmeans that the worst model built by applying machine learningtechniques triplicates the possibilities of suggesting a suitablecomponent to users when they need it, in the case that compo-nents were to be chosen randomly.

In our case study, there is a class imbalance, i.e., the mostfrequent case GeoParks has six times the number of occur-rences than the less frequent case Clock2, as previously shownin Table 4. The class imbalance is not extremely high but itneeds to be considered. For this reason, the Overall Accu-racy measure is not enough, we also need the Average Accu-racy measure to really determine the performance of the mod-els. The Overall Accuracy indicates the correctly predicted in-stances (number of correctly predicted items / total of items topredict). The Average Accuracy measures the ability to predictclasses with few occurrences, difficult to predict (sum of the ac-curacy for each class predicted / number of classes). Table 6presents both measures and we can see that given the perfor-mance of the Average Accuracy the models created using theDecision Forest and Decision Jungle algorithms have problemspredicting the less frequent classes and the Artificial Neural-Networks and Logistic Regression models behave more consis-tently across all classes to predict.

Data can be analyzed in more detail by studying the Con-fusion Matrix. A Confusion Matrix is a table layout whereeach row represents the number of instances of each class andeach column represents the class that has been predicted by themodel. In this table, we can obtain a reliable performance ofthe model beyond the global accuracy. We have created a cus-tomized Confusion Matrix (Table 7) that has been divided intotwo parts. The upper part shows the results of the models cre-ated using the whole dataset and the lower part shows the re-sults of the models created using a subset of features. Each cellshows 4 values, corresponding to each algorithm used.

The detailed values of the accuracy of each model predict-ing each class are shown in Table 7. We have also created asurface chart representing the Confusion Matrix, shown in Fig-ure 7, that helps to easily visualize the results thrown by themodels created. As can be noted from this chart, the Deci-sion Forest and Decision Jungle algorithms, regardless of usingthe whole dataset or a subset of it, show a high concentrationof occurrences that are not correctly predicted at the bottompart of their chart. The bottom part of the graphic correspondsto the instances where the predicted class is GeoParks, themost common label. Additionally, the Decision Jungle algorit-hms have serious problems predicting the Weather, Heritageand Clock2 components. It looks like these algorithms tendto predict the most common label GeoParks. Conversely, theArtificial Neural Networks and Logistic Regression algorithmspresent better results, specifically the model created using Ar-tificial Neural Networks is particularly suitable for creating therecommendation system that we aim to create in this research.The surface chart shows that this model is very consistent ac-curately predicting all classes as well as being highly accuratepredicting the most uncommon classes. These attributes ensure

Figure 7: Surface Chart representing the Confusion Matrix

11

BR C2 CR GP HT TW WT WL

BR

0.72 0.00 0.00 0.25 0.02 0.00 0.00 0.00 DF0.58 0.00 0.11 0.25 0.00 0.00 0.04 0.00 DJ1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 ANN0.75 0.00 0.00 0.19 0.00 0.00 0.06 0.00 LR

C2

0.07 0.52 0.00 0.34 0.00 0.00 0.07 0.00 DF0.00 0.22 0.22 0.56 0.00 0.00 0.00 0.00 DJ0.12 0.75 0.00 0.00 0.00 0.12 0.00 0.00 ANN0.07 0.47 0.00 0.33 0.13 0.00 0.00 0.00 LR

CR

0.00 0.00 0.67 0.26 0.03 0.03 0.03 0.00 DF0.00 0.00 0.67 0.33 0.00 0.00 0.00 0.00 DJ0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 ANN0.00 0.00 0.42 0.24 0.15 0.00 0.15 0.03 LR

GP

0.00 0.00 0.04 0.90 0.02 0.02 0.03 0.00 DF0.00 0.00 0.02 0.97 0.00 0.00 0.01 0.00 DJ0.00 0.04 0.00 0.92 0.04 0.00 0.00 0.00 ANN0.02 0.00 0.00 0.97 0.02 0.00 0.00 0.00 LR

HT

0.00 0.00 0.00 0.60 0.39 0.00 0.01 0.00 DF0.00 0.00 0.01 0.85 0.11 0.00 0.03 0.00 DJ0.00 0.00 0.00 0.09 0.87 0.00 0.04 0.00 ANN0.00 0.00 0.02 0.22 0.72 0.00 0.04 0.00 LR

TW

0.00 0.00 0.09 0.47 0.07 0.36 0.02 0.00 DF0.00 0.00 0.11 0.82 0.00 0.08 0.00 0.00 DJ0.00 0.05 0.00 0.10 0.15 0.70 0.00 0.00 ANN0.00 0.00 0.13 0.13 0.00 0.71 0.03 0.00 LR

WT

0.02 0.00 0.13 0.37 0.05 0.05 0.40 0.00 DF0.00 0.00 0.14 0.79 0.00 0.00 0.07 0.00 DJ0.00 0.00 0.07 0.10 0.13 0.07 0.60 0.03 ANN0.04 0.00 0.00 0.10 0.10 0.00 0.76 0.00 LR

WL

0.00 0.00 0.07 0.00 0.00 0.00 0.00 0.93 DF0.00 0.00 0.32 0.00 0.00 0.00 0.00 0.68 DJ0.00 0.00 0.18 0.00 0.18 0.00 0.00 0.64 ANN0.00 0.00 0.07 0.29 0.07 0.00 0.07 0.50 LR

(a) Without Feature Selection

BR C2 CR GP HT TW WT WL

BR

0.81 0.00 0.00 0.19 0.00 0.00 0.00 0.00 DF0.63 0.00 0.07 0.30 0.00 0.00 0.00 0.00 DJ1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 ANN0.75 0.00 0.00 0.13 0.00 0.00 0.13 0.00 LR

C2

0.00 0.72 0.00 0.28 0.00 0.00 0.00 0.00 DF0.00 0.22 0.25 0.50 0.00 0.00 0.00 0.03 DJ0.25 0.75 0.00 0.00 0.00 0.00 0.00 0.00 ANN0.07 0.60 0.00 0.27 0.07 0.00 0.00 0.00 LR

CR

0.00 0.00 0.62 0.36 0.00 0.00 0.03 0.00 DF0.00 0.00 0.69 0.27 0.00 0.00 0.00 0.04 DJ0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 ANN0.00 0.00 0.52 0.18 0.06 0.03 0.15 0.06 LR

GP

0.00 0.00 0.00 0.94 0.02 0.01 0.03 0.00 DF0.00 0.00 0.03 0.96 0.01 0.00 0.00 0.00 DJ0.00 0.00 0.00 0.96 0.04 0.00 0.00 0.00 ANN0.02 0.00 0.00 0.98 0.00 0.00 0.00 0.00 LR

HT

0.00 0.00 0.00 0.55 0.39 0.01 0.05 0.00 DF0.00 0.00 0.02 0.74 0.22 0.00 0.01 0.01 DJ0.04 0.00 0.00 0.09 0.87 0.00 0.00 0.00 ANN0.00 0.00 0.04 0.28 0.64 0.00 0.04 0.00 LR

TW

0.00 0.00 0.07 0.51 0.00 0.38 0.04 0.00 DF0.00 0.00 0.14 0.80 0.00 0.06 0.00 0.00 DJ0.00 0.00 0.00 0.25 0.05 0.70 0.00 0.00 ANN0.00 0.00 0.08 0.08 0.00 0.76 0.08 0.00 LR

WT

0.00 0.00 0.02 0.51 0.00 0.00 0.48 0.00 DF0.00 0.00 0.11 0.74 0.00 0.00 0.14 0.01 DJ0.00 0.00 0.07 0.17 0.07 0.07 0.63 0.00 ANN0.04 0.00 0.00 0.12 0.08 0.00 0.76 0.00 LR

WL

0.00 0.00 0.14 0.00 0.00 0.00 0.00 0.86 DF0.00 0.00 0.23 0.14 0.00 0.00 0.00 0.64 DJ0.00 0.00 0.18 0.00 0.18 0.00 0.00 0.64 ANN0.00 0.00 0.00 0.36 0.00 0.00 0.00 0.64 LR

(b) With Feature Selection

Table 7: Confusion Matrix (Components {BR: BioReserves; C2: Clock2; CR: CuttleRoads; GP: GeoParks; HT: Heritage; TW: Twitter; WT: Weather; WL:Wetlands}— ML Algorithms {DF: Decision Forest; DJ: Decision Jungle; ANN: Artificial Neural Network; LR: Logistic Regression})

Figure 8: Recommendation model output for a concrete scenario in ENIA

12

that the model is reliable and we can conclude that the modelcreated using the Artificial Neural Networks, trained with a sub-set that contains the most significant features extracted usingthe Mutual Information Score and Chi-Squared Statistic Fea-ture Selection methods, is the best model created to suggest themost suitable component to users in component apps.

5.3. DeploymentOnce the best model is selected, it has to be deployed in the

component-based application. The real output of the recom-mendation system is the list of possible components to suggest,along with the grade of certainty of the recommendation, ex-pressed as a percentage.

In order to avoid the risk of forecasting a component thatis not good for a user in a concrete scenario, in ENIA we donot directly suggest the most suitable component to a user ac-cording to the output of the model. Instead, we evaluate if thelikelihood of the component suggested is higher than 50%. Ifso, the component will be suggested to the end-user.

Figure 8 illustrates the output of the model for a concretescenario in the ENIA component-based web application. Thescored label is the class with higher probability, in this case,the Wetlands component, is the class with the higher score(0.544146). As the likelihood of this class is higher than 50%we proceed to suggest the Wetlands component to the end-userin this scenario.

6. Conclusions and Future Work

In this article, we address the problem of creating a recom-mender system that is able to suggest to users of component-based interfaces, which are the most suitable components forthem to use at a specific time. By forecasting the componentmost closely aligned to each situation, we aim to improve theuser experience in the software application and thus, optimizethe possibilities of successfully achieving a good position in theincreasingly competitive software development market.

We have created several models using a dataset that con-tains the interactions performed by users in component basedapplications after applying Feature Engineering techniques andFeature Selection methods. After this, we have evaluated andcompared the models created in terms of overall accuracy andaverage accuracy, as well as analyzing the confusion matrix thatoffers the specifying accuracy including the deviation that themodel may have predicting a class compared with the labeledclass. As a conclusion, all the recommender systems get goodresults achieving an accuracy of up to 40%, with the NeuralNetwork models being the one that gets better performance,yielding an accuracy of up to 80%. It is remarkable that FeatureSelection methods favorably affect to the results, increasing theaccuracy by an average of 3% in all cases.

In future works, it would be interesting to improve the rec-ommendation of components in component-based user inter-faces by following these practices:

a) Instead of using just the recommendation model with high-est accuracy, the creation of a network that agglutinates

all the models and votes the most popular output of eachof them, considering the normalized accuracy of each,could improve the overall performance.

b) Since there are components that are more frequently usedthan others, with a huge difference, adding weights thathelp to better forecast the less common classes would im-prove the average accuracy of the recommendation model.

c) Given the rapid evolution of users and components incomponent-based user interfaces and in the software de-velopment market, a nice improvement would be the def-inition of a coherent strategy to continuously update therecommendation model so it can be adapted to the changesin the software application environment.

Through the implementation of these improvements and thecontinuous optimization of the dataset, adding more instancesand significant features, the recommendation system can beproperly optimized over time.

Acknowledgements

This work has been funded by the EU ERDF and the SpanishMinistry of Economy and Competitiveness (MINECO) under ProjectsTIN2013-41576-R and TIN2017-83964-R. A.J. Fernandez-Garcıa hasbeen funded by a FPI Grant BES-2014-067974. J.Z. Wang was fundedby the US National Science Foundation under Grant No.1027854.

References

[1] ENIA. Environmental Information Agent Project. http://acg.ual.

es/projects/enia/ui/. Online; last accessed 18 December 2017.[2] Open Weather Map. https://openweathermap.org/. Online; last

accessed 18 December 2017.[3] The Andalusian Environmental Information Network (REDIAM). http:

//www.juntadeandalucia.es/medioambiente/site/rediam/.Online; last accessed 18 December 2017.

[4] Salem Alelyani, Jiliang Tang, and Huan Liu. Feature Selection for Clus-tering: A Review. Chapman&Hall, 2013.

[5] Dheeraj Bokde, Sheetal Girase, and Debajyoti Mukhopadhyay. Matrixfactorization model in collaborative filtering algorithms: A survey. Pro-cedia Computer Science, 49(Supplement C):136 – 146, 2015.

[6] Leo Breiman. Bagging predictors. Machine Learning, 24(2):123–140,Aug 1996.

[7] Pedro A. Castillo, Antonio M. Mora, Hossam Faris, J.J. Merelo, PabloGarcıa-Sanchez, Antonio J. Fernandez-Ares, Paloma De las Cuevas, andMarıa I. Garcıa-Arenas. Applying computational intelligence methods forpredicting the sales of newly published books in a real editorial businessmanagement environment. Knowledge-Based Systems, 115(SupplementC):133 – 151, 2017.

[8] Yehonatan Cohen, Danny Hendler, and Amir Rubin. Detection of mali-cious webmail attachments based on propagation patterns. Knowledge-Based S., 141:67 – 79, 2018.

[9] Antonio Jesus Fernandez-Garcıa, Luis Iribarne, Antonio Corral, JavierCriado, and James Z. Wang. Optimally storing the user interaction inmashup interfaces within a relational database. In Current Trends in WebEngineering, pages 188–195, Cham, 2016. Springer Int. Pub.

[10] Antonio Jesus Fernandez-Garcia, Luis Iribarne, Antonio Corral, andJames Z. Wang. Evolving mashup interfaces using a distributed ma-chine learning and model transformation methodology. In On the Moveto Meaningful Internet Systems, pages 401–410. Springer Int. Pub., 2015.

[11] Eva-Patricia Fernandez-Manzano, Elena Neira, and Judith Clares-Gavilan. Data management in audiovisual business: Netflix as a casestudy. Profesional de la Informacion, 25(4):568–576, 2016.

13

[12] Antonio Jesus Fernandez-Garcıa, Luis Iribarne, Antonio Corral, JavierCriado, and James Z. Wang. A flexible data acquisition system for stor-ing the interactions on mashup user interfaces. Computer Standards &Interfaces, 2018. (In press).

[13] FitBit. FitBit. Fitness products family to monitor activity, exercise, food,weight and sleep. https://www.fitbit.com, 2018.

[14] Flipboard. Flipboard. News and Social Media Aggregation Platfrom.https://flipboard.com/, 2018.

[15] Ines M. Galvan, Jose M. Valls, Alejandro Cervantes, and Ricardo Aler.Multi-objective evolutionary optimization of prediction intervals for so-lar energy forecasting with neural networks. Information Sciences, 418-419:363 – 382, 2017.

[16] Salvador Garcia, Julian Luengo, Jose A. Saez, Victoria Lopez, and Fran-cisco Herrera. A survey of discretization techniques: Taxonomy and em-pirical analysis in supervised learning. IEEE Trans. on Knowl. and DataEng., 25(4):734–750, April 2013.

[17] Lise Getoor and Ben Taskar. Introduction to Statistical Relational Learn-ing (Adaptive Computation and Machine Learning). The MIT Press,2007.

[18] Girish Chandrashekar and Ferat Sahin. A survey on feature selectionmethods. Computers & Electrical Engineering, 40(1):16 – 28, 2014.

[19] Google. Google Now. Personal Assistant. https://www.google.com/intl/es/landing/now/, 2018.

[20] Sergio Gomez, Panagiotis Zervas, Demetrios G. Sampson, and RamonFabregat. Context-aware adaptive and personalized mobile learning de-livery supported by uolmp. Journal of King Saud University - Computerand Information Sciences, 26(1, Supplement):47 – 61, 2014.

[21] Petr Hajek and Roberto Henriques. Mining corporate annual reports forintelligent detection of financial statement fraud – comparative study ofmachine learning methods. Knowledge-Based Syst., 128:139 – 152, 2017.

[22] Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The elements ofstatistical learning: Data mining, inference, and prediction. InternationalStatistical Review, 77(3):482–482, 2009.

[23] Mehmed Kantardzic. Data Mining: Concepts, Models, Methods and Al-gorithms. John Wiley & Sons, Inc., 2002.

[24] Wei-Po Lee, Chun-Ting Chen, Jhih-Yuan Huang, and Jhen-Yi Liang. Asmartphone-based activity-aware system for music streaming recommen-dation. Knowledge-Based Systems, 131(Supplement C):70 – 82, 2017.

[25] Alex Liu. Javascript and the netflix user interface. Commun. ACM,57(11):53–59, October 2014.

[26] Nathan Mantel. Chi-square tests with one degree of freedom; extensionsof the mantel-haenszel procedure. J. of the American Statistical Assoc.,58(303):690–700, 1963.

[27] P. McCullagh and J.A. Nelder. Generalized Linear Models, Second Edi-tion. Chapman and Hall/CRC Monographs on Statistics and AppliedProbability Series. Chapman & Hall, 1989.

[28] Microsoft Corporation. Microsoft Azure Machine Learning Studios.[29] Netflix. Netflix. A component-based video-on-demand streaming plat-

form. https://www.netflix.com/, 2018.[30] Oracle Corporation. MySQL Database.[31] Ivens Portugal, Paulo Alencar, and Donald Cowan. The use of machine

learning algorithms in recommender systems: A systematic review. Ex-pert Systems with Applications, 97:205 – 227, 2018.

[32] J. R. Quinlan. Induction of decision trees. Mach. Learn., 1(1):81–106,March 1986.

[33] Karthik Ramasubramanian and Abhishek Singh. Feature Engineering,pages 181–217. Apress, Berkeley, CA, 2017.

[34] Raul Rojas. Neural Networks: A Systematic Introduction. Springer-Verlag New York, Inc., 1996.

[35] C. Sanchez, N.M. Villegas, and J. Dıaz Cely. Exploiting context infor-mation to improve the precision of recommendation systems in retail-ing. Communications in Computer and Information Science, 735:72–86,2017.

[36] J. Vallecillos, J. Criado, N. Padilla, and L. Iribarne. A cloud service forCOTS component-based architectures. Computer Standards& Interfaces,48:198–216, 2016.

[37] Jesus Vallecillos, Javier Criado, Antonio Jesus Fernandez-Garcıa, NicolasPadilla, and Luis Iribarne. A web services infrastructure for the manage-ment of mashup interfaces. In Service-Oriented Computing – ICSOC2015 Workshops, pages 64–75. Springer, 2016.

[38] Norha M. Villegas, Cristian Sanchez, Javier Dıaz-Cely, and Gabriel

Tamura. Characterizing context-aware recommender systems: A system-atic literature review. Knowledge-Based Systems, 140:173 – 200, 2018.

[39] Shaoqing Wang, Cuiping Li, Kankan Zhao, and Hong Chen. Learning tocontext-aware recommend with hierarchical factorization machines. In-formation Sciences, 409-410:121 – 138, 2017.

[40] Zexian Wei, Yanxue Wang, Shuilong He, and Jia Bao. A novel intelligentmethod for bearing fault diagnosis based on affinity propagation cluster-ing and adaptive feature selection. Knowledge-Based Syst., 116:1 – 12,2017.

[41] Ke Xu, Xushen Zheng, Yi Cai, Huaqing Min, Zhen Gao, Benjin Zhu,Haoran Xie, and Tak-Lam Wong. Improving user recommendation byextracting social topics and interest topics of users in uni-directional so-cial networks. Knowledge-Based Systems, 140(Supplement C):120 – 133,2018.

[42] Shuo Yang, Mohammed Korayem, Khalifeh AlJadda, Trey Grainger, andSriraam Natarajan. Combining content-based and collaborative filter-ing for job recommendation system: A cost-sensitive statistical relationallearning approach. Knowledge-Based Systems, 136(Supplement C):37 –45, 2017.

[43] Yudong Zhang, Shuihua Wang, Preetha Phillips, and Genlin Ji. Binarypso with mutation operator for feature selection using decision tree ap-plied to spam detection. Knowledge-Based Systems, 64:22 – 31, 2014.

[44] Yao Zhou, Hua Mao, and Zhang Yi. Cell mitosis detection using deepneural networks. Knowledge-Based Systems, 137:19 – 28, 2017.

14