A Visual Programming Paradigm for Abstract DeepLearning Model Development

Srikanth Tamilselvam,† Naveen Panwar,† Shreya Khare,†Rahul Aralikatte,‡ Anush Sankaran,† Senthil Mani,†

†IBM Research, ‡University of Copenhagen

ABSTRACTDeep learning is one of the fastest growing technologies incomputer science with a plethora of applications. But this un-precedented growth has so far been limited to the consumptionof deep learning experts. The primary challenge being a steeplearning curve for learning the programming libraries and thelack of intuitive systems enabling non-experts to consumedeep learning. Towards this goal, we study the effectivenessof a “no-code" paradigm for designing deep learning models.Particularly, a visual drag-and-drop interface is found moreefficient when compared with the traditional programmingand alternative visual programming paradigms. We conductuser studies of different expertise levels to measure the entrylevel barrier and the developer load across different program-ming paradigms. We obtain a System Usability Scale (SUS)of 90 and a NASA Task Load index (TLX) score of 21 forthe proposed visual programming compared to 68 and 52,respectively, for the traditional programming methods.

ACM Classification KeywordsD.1.7 PROGRAMMING TECHNIQUES (E): Visual Pro-gramming; I.2.2 ARTIFICIAL INTELLIGENCE: AutomaticProgramming—Program synthesis; D.2.2 SOFTWARE EN-GINEERING: Design Tools and Techniques—User interfaces

Author KeywordsDeep Learning, Visual Programming, drag-and-drop interface,auto programming

INTRODUCTIONDeep learning (DL) is one of the most pervasive and break-through technologies of the previous decade [26]. Automationhas progressed significantly in various application domainssuch as self-driving cars [12] [13] [19], flight control systems[28], and medical diagnosis [22] [16] primarily due to the ad-vent of deep learning technology. According to Gartner, 80%of the data scientists will have DL in their toolkits by 2018 1.With this sudden explosion of DL, there is a requirement forcurrent software engineers and data scientists to learn and up-skill themselves in a short span of time. However, DL modelauthoring has been a skill that is currently restricted only toexperts 2. The democratization of DL model development hasbeen inhibited due to the following challenges:

1. High Initial Learning Curve: There is a steep learningcurve involved in understanding the fundamentals of deep

1https://www.gartner.com/newsroom/id/38043632https://www.kdnuggets.com/2018/01/democratizing-ai-deep-learning-machine-learning-dell-emc.html

learning and the syntax nuances of multiple deep learningauthoring libraries. There is a lack of intuitive systems toauthor DL models in an easy fashion.

2. Lack of Interaction across Libraries: Different program-ming libraries exist for DL implementation such as Keras3,Tensorflow4, PyTorch5, Caffe6. However, there is limitedinteroperability across these libraries that enables a modeldesigned and trained in one specific library to be accessibleand re-usable across libraries.

3. Theoretical Knowledge: Theory of deep learning has astrong mathematical prior. Additionally, the ability of mak-ing the right architecture choices and hyper-parameter pref-erences comes with practice and experience.

Sankaran et al. [25] studied the challenges faced by a DL devel-oper by conducting a qualitative survey among 100 softwareengineers from varying backgrounds. 83% of the participantsresponded that it took them about 3−4 days to implement adeep learning model, given the model design and choice to useany DL library. Interestingly, 86% of those respondents hadrated themselves as highest in “programming ability". Thus,even good programmers find it challenging to implement andauthor deep learning models. This motivates the need for anefficient and intuitive method for developing deep learningmodels. The survey also showed that more than 92% of therespondents wanted an interoperability framework to convertthe code and model implemented in one library into anotherlibrary. In the recent past, a few open source unified librarieswere made available in the community such as Open NeuralNetwork Exchange (ONNX) [10], nGraph from Intel [4], andNeural Network Exchange Format (NNEF) [8]. These unifiedframeworks provide a standard, library agnostics represen-tation of deep learning models. Despite the availability ofsuch standardized frameworks, the ease in adoption of DL fornon-expert users and rapid prototyping for expert users stillremains a challenge.

In this research, we develop an easy to use visual programmingparadigm and an Integrated Development Environment (IDE)for intuitive designing and authoring of deep learning models.This DL-IDE aims to democratize deep learning model devel-opment for software engineers and data scientists by offering a“no-code" designing platform and reducing the learning curve.

3https://keras.io/4https://www.tensorflow.org/5https://pytorch.org/6https://caffe2.ai/

arX

iv:1

905.

0248

6v2

[cs

.HC

] 1

9 A

ug 2

019

Tool UIDesigning

ModelVisualization

CodeAvailable

Multi-librarysupport

TrainingDashboard

InferenceAPI

Weka [17] Limited Limited Yes No Yes YesDigits [9] No Yes No No Yes NoAetros [3] Yes No Yes No Yes NoFabrik [1] Yes Yes Yes Yes No NoTensorboard [11] No Yes No No Yes NoAzure ML No No No No Yes NoNN Console [7] Yes Yes No No Yes NoActivis [20] No Yes No No No NoNetron [5] No Yes No No No NoDeep Cognition [2] Yes Yes No No Yes YesMachine UI [6] Yes Yes Yes No Yes NoDL-IDE (proposed) Yes Yes Yes Yes Yes Yes

Table 1: Comparison of different tools and frameworks in the literature that enables easy and quick development of deep learningmodels.

Fundamentally, any deep learning model design can be vi-sualized as a graph data structure with a collection of nodesas computational layers and each layer having a list of hyper-parameters. To ease designing of such models, the DL-IDE hasa drag-and-drop framework to visually design the deep learn-ing model. The layer connections and the layer parameterscould be set through the user interface minimizing the needfor coding. DL-IDE represents the designed deep learningmodel as an abstract representation called Neural LanguageDefinition Standard (NLDS) enabling execution ready code tobe generated in multiple libraries such as Tensorflow, Keras,PyTorch, and Caffe. To study the effectiveness of the proposedDL-IDE, we conducted usability study among 18 intermediateand expert deep learning developers, comparing the visualmethod and programming method for designing deep learn-ing models. The primary aim of the user study is to verifythe hypothesis that, in comparison with the traditional codingmethod for designing deep learning models, the visual pro-gramming [23] method is better in terms of ease-of-use, easein adoption, and decreases the prototyping efforts. The keyresearch contributions can be summarized as follows:

1. Visual DL-IDE: A visual programming IDE enabling “no-code" intuitive way of designing deep learning models7.

2. Neural Language Definition Standard (NLDS): An ab-stract representation capturing all the required parametersof a DL model, independent of the underlying implementa-tion library (Tensorflow, PyTorch etc.) or language.

3. Static Model Validation: An extensive collection of man-ually curated rules to automatically validate the design pa-rameters of a deep learning model in real-time, performedon top of NLDS.

7To abide to the double blind format of the paper, only screenshotsof the system is made available. The entire system will be madepublicly available upon the acceptance of the paper.

4. Multiple Library Code Generation: Adopting ModelDriven Development of generating bug free, execution readycode in multiple libraries from NLDS representation. Cur-rently, we support Tensorflow v1.4, Keras v2.1, PyTorchv0.3, and Caffe v1.0.

5. System Usability Study: A user study is conducted with18 intermediate and expert deep lear ning developers toassess, and compare the traditional programming methodwith the proposed visual drag-and-drop user interface.

The rest of the paper is organized as follows: Section 2 pro-vides a literature survey comparing the DL-IDE with otherabstract visual programming tools. Section 3 details the de-sign goals and the system architecture of the proposed deeplearning IDE. Section 4 explains the user study setup and thedata collection process. Section 5 analyzes the experimentalresults from the user study. Section 6 discusses the designimplications of our study setup, and Section 7 concludes thisresearch work discussing some potential future extensions.

RELATED WORKThe main requirements for any deep learning tool to be usablecan be summarized into three major categories:

• Design and Visualization: This includes features such asintuitive designing and construction of deep learning mod-els, visualization of pre-designed models, and dashboardingof model logs during training.

• Elimination of need to code: This includes features suchas automatic source code generation, support for multiplelibraries, and import from existing codebases.

• Production-ready capabilities: This includes featuressuch as efficiently training the model, support for easy in-ference, and fast and scalable deployment.

Most of today’s tools are aimed at addressing one or at mosttwo of these categories. Table 1 summarizes the various tools

features. The concept of easy designing of models dates backto Weka [17]. Weka was “no-code" method for designing tradi-tional machine learning tasks having limited support for neuralnetworks. It provides a dashboard for visualizing training met-rics and a Java-based API for inference. NVIDIA DIGITS [9]was the first deep learning specific tool to provide an userinterface to perform tasks like data loading and one-click train-ing. However, it is only possible to train pre-built networksand it does not have the capability to design models fromscratch. More recently, tools like Google’s Tensorboard [11]is often used as a supplement to visualize Tensorflow modelsand Netron [5] is used to visualize existing models coded invarious libraries.

Though these visualization tools help, there is still a needto visually author a new model from scratch and not justvisualize an existing model. There is a need for an end-to-endsystem aimed to seamlessly integrating the design and trainingprocedures. Sony’s Neural Network Console [7], Aetros [3],Deep Cognition [2], Fabrik [1], and Machine UI [6] providean intuitive drag-and-drop based interface to design customdeep learning models. They also have a compute backend totrain the designed models and present the training metrics in areal-time dashboard. Although, new models could be designedusing the drag-and-drop interface, most of these do not allowthe corresponding execution ready code to be downloaded.These tools use their own compute backend for training butusers often refuse to use such tools as uploading confidentialdata into these third party services is conflicting.

Thus it can be observed that, though there exists multiple toolswhich perform one or more tasks sufficiently, there is a require-ment to address the problem as a whole with an end-to-endperspective. Therefore, we created DL-IDE which supportsvisual authoring, generation of code in multiple existing li-braries, and download design (NLDS) as well as code for DLmodel development.

DEEP LEARNING IDE (DL-IDE) SYSTEM ARCHITECTURESoftware developers implement deep learning models by fol-lowing the DL architecture details documented in researchpapers, blogs among others. Commonly, these deep learn-ing models and its parameters are expressed either as a flowdiagram or in a tabular format, as shown in Figure 1a and1b. These architectural information are transformed into aprogram (code) by the developers. Currently, this coding taskis time taking, laborious, and requires expert level skill [25].Motivated from these challenges, our aim is to design an IDEfor deep learning where models are implemented in the sameway as it is represented in documents and research papers;visually. The design goals of the proposed DL-IDE would beas follows:

1. A visual “no-code" designer to reduce the entry level barrierfor DL programming.

2. A drag-and-drop interface to design DL models as a combi-nation of nodes (layers) and connections, corresponding tovisual designs represented in research papers.

3. A real-time validation service to verify the designed modeland provide specific feedback for easy debugging

(a) Flow figure [27]

(b) Tabular format [24]

Figure 1: Two popular methods for expressing deep learningmodel design in research papers, presentations, and documents.Developers typically implement the models by following theserepresentations.

4. An abstract DL-library independent representation of deeplearning model, to enable interoperability across librariesand platforms.

5. Export executable code from the visually constructed DLmodel in the library of user’s choice.

The DL-IDE system, shown in Figure 2, consist of four maincomponents: (i) visual drag-and-drop designer, (ii) abstractintermediate representation generator, (iii) a real-time modeldesign validator, and (iv) code generator in multiple libraries.

Visual DesignerUser interacts with our system through the designer com-ponent, which consists of two sub-components. The “DL-modeler" sub-component provides a drag-and-drop canvas forthe user to author a DL model. It also provides a palette ofavailable layers on the left and a section for displaying a de-tailed documentation on the right as shown in Figure 4. Thispalette groups layers performing similar function to ease devel-opment of a model for a new user. A user can drag drag layers,configure its properties, and connect them to create a deepneural network in a language/library agnostic manner. Whena user selects a layer from the palette, the system providesan initial default configuration of the parameters, which usercan update based on the design needs. The system performsreal-time validation of the model being designed and displaysmessages/errors and potential solutions on the right side sec-tion of the “DL-modeler". The “Dashboard" sub-componentprovides a real-time view of the model’s training progress.The user can choose the metrics to be plotted as a graph in thedashboard, view various parameters, and hyper-parameters ofthe training experiment along with visualization of trainingdata.

Figure 2: The overall system architecture and the components of the proposed deep learning IDE, DL-IDE.

Figure 3: An example of the Neural Layer Definition Standard (NLDS) for the popular LSTM layer, along with their correspondingCaffe Prototxt and Keras python code.

Figure 4: The visual drag-and-drop canvas user interface of the DL-IDE.

Neural Layer Definition StandardA deep learning model is a sequential or non-sequential col-lection of different transformations performed on the inputdata to obtain the output. Each transformation, called as alayer, is the fundamental building block of a deep learningmodel. Neural Language Definition Standard (NLDS) is astandard, abstract representation of deep learning model thatis agnostic of the underlying library. The NLDS definition ofa deep neural network is a JSON object contains three mainparts:

• Layer definitions: defines the parameters of every layer inthe network.

• Layer links: defines how the layers are connected to eachother.

• Hyperparameter definitions: defines model level param-eters which define some characteristic properties of themodel as a whole.

These three parts of the NLDS captures all aspects of a model’sdesign. Individual library transformers can be written on top ofthe NLDS definition to import and export execution ready codeacross multiple libraries. For example, a tensorflow code couldbe parsed to extract the NLDS definition of the DL model, us-ing which a PyTorch code can be generated. Thus, a losslessconversion could be performed across libraries through NLDS,as a model driven development strategy. An example NLDSfor the popular LSTM layer and its corresponding Caffe Pro-totxt and Keras python code is shown in Fig. 3.

DL Static Design ValidationThe validation component, validates the model being con-structed in the “DL-modeler". For every update of the DLmodel design in the drag-and-drop interface, the validationcomponent checks the correctness of the overall model. Anywarnings or errors are displayed as a pop-up on the validationsection of the user interface. User cannot progress to the nextstages of code download or training, if the constructed modelis not successfully validated. Validation of a DL design hap-pens in five levels: (1) basic NLDS structure validation, (2)layer level parameter validation, (3) static design validationfor ensuring valid next and previous layers for every layer,(4) flow design validation, ensuring the data flow through themodel, and (5) platform (target library) specific validation. Forexample, if a Convolution2D layer is added without providingthe number of filters, the validation component would promptto provide the compulsory parameters. If a Dense layer isadded after Convolution2D layer, the validation componentwould prompt that a Dense could not be added after a Convo-lution 2D layer, and also recommend a solution of adding aflatten layer in between.

Multiple Library Code GenerationOnce the constructed model is validated, the user can eitheropt to generate and download the source code of the modelin a desired library or push the model for training. In thefirst case, the source code for the model is generated in thelibrary of user’s choice for download. This gives the flexibilityto the user to use this tool for authoring the model in theirpreferred library of choice and train them in the environmentof their choosing. In the second case, the model is converted

into the preferred library’s code and pushed to the defaultback-end GPU cluster for training. Currently, the tool supportthree libraries namely TensorFlow, Keras, Pytorch for codegeneration.

The GPU cluster can be considered as a pluggable extension tothe IDE and is not one of its integral part as it can be replacedby any external cloud GPU service such as AWS8 or GoogleCloud9. The main purpose of this component is to train themodel on the data provided by the user. The “execution en-gine" of this component contains GPU hardware and wrappersfor low-level firmware like CUDA which come out-of-the-box.The “log manager" is a custom component which continuouslymonitors the training logs for certain metrics identified by theuser. These metric values are streamed to the UI dashboardand plotted as graphs. After the training is completed, thetrained model is deployed and an API is exposed to the userfor inference.

DESIGNING USER STUDYTo study the usability and the efficiency of the proposed visualdrag-and-drop programming paradigm for DL, we conducteda user study to compare the proposed system with traditionalprogramming paradigm and other alternative visual program-ming paradigms.

Different Programming ParadigmsIt can be observed from Figure 1b that a DL model’s designcould be represented in a tabular format, as well. Hence, itis worthwhile to study whether a visual tabular interface isefficient and easy for authoring DL models. To compare DL-IDE with the tabular representation of designing deep learningmodels, a visual tabular interface is designed as shown inFigure 5. Overall in the user study, the participants were giventhree different views to implement deep learning models:

1. View #1 - Code: Traditional coding method for imple-menting deep learning models. As a common standard,Microsoft Visual Studio Code 10 is used as the Python IDE.The participants were allowed to use the DL library thatthey are most fluent with, among the following: Keras,Tensorflow, PyTorch, Caffe.

2. View #2 - Tabular: Tabular way of designing a deep learn-ing model, as shown in Figure 5.

3. View #3 - DL-IDE (drag-and-drop): The proposed sys-tem to design a deep learning model like a graph structureusing a drag-and-drop interface, as shown in Figure 4.

To avoid any bias, the participants had no prior on the motiveof the experiments or on the view of interest.

ParticipantsThe user study was conducted among 18 participants who arestudents from different universities. All the participants arerequired to have basic knowledge of developing deep learningmodels, in addition, to being efficient in python programming.8https://aws.amazon.com/9https://cloud.google.com/

10https://code.visualstudio.com/

Further, none of the participants had any prior affiliation withany of the authors and the purpose of the study remainedanonymous.

The participants filled an qualitative survey providing demo-graphic details and their technical competence along withDL preferences. The participants comprised of 11 male and7 female participants whose age range is between 19− 24.Furthermore, the participants were highly comfortable withPython programming language and 15 participants rated them-selves as 3 or more out of 5 in deep learning implementation.

To better characterise the outcome of our study, the 18 partic-ipants are divided either as expert or intermediate based ontheir answers to the following eight questions from the survey:

(A) Are you aware of Python language ? Ans: Yes/No

(B) How would you rate your comfort in Python? Scale: 1-5

(C) How would would rate your comfort in implementing DeepLearning algorithms? Scale: 1-5

(D) Have you designed Deep Learning models as part of aproject and/or paper ? Ans: Yes/No

(E) How many deep learning models have you designed?

(F) How comfortable are you in designing Deep Learning?Scale: 1-5

(G) Have you come across Deep Learning model in researchpublications (papers)? An: Yes/No

(H) How many deep learning model designs have you comeacross in research publications (papers) ?

An expertise score, S, is calculated for every user using all theabove eight information, with the equation below,

Expertise Score,S = A∗B+D∗ (C+F)∗E +G∗H (1)

The median of the expertise score is computed and all thoseparticipants above the median score are labeled experts whilethose below are labeled intermediate. The expertise scoreranged between 19−264 with a median of 36. There were intotal 9 expert and 9 intermediate users in our study.

Experimental ProcedureThe experiment involves participating users to implement threedeep learning model designs. Details of these models are listedbelow as three tasks:

1. Task #1: Implementing a 13 layer deep convolutional neu-ral network (CNN) model on ImageNet [15] image datasetwith Conv2D, Pool2D, TanH, ReLU, Flatten, Dense, andSoftmax as the set of unique layers.

2. Task #2: Implementing a 16 layer deep convolutional neu-ral network (CNN) model on CIFAR-10 [21] image datasetwith Conv2D, Pool2D, ReLU, Flatten, Dense, and Softmaxas the set of unique layers.

3. Task #3: Implementing a 6 layer deep recurrent neuralnetwork (RNN) model on text classification dataset with

Figure 5: The tabular representation based UI system to design deep learning models.

Embedding, LSTM, Dense, and Softmax as the set of uniquelayers.

To avoid any known model bias, popular deep learning modelswere ignored for this study and the above three novel modelswere designed from scratch. Based on the author’s experiencein deep learning, they are also of equal difficulty level inimplementation.

Each participant was asked to perform three experiments: Ex-periment 1, Experiment 2, and Experiment 3 where, in eachexperiment, they implemented one of the three tasks in one ofthe three views. To counter for the learning effect, the task-view combination for all 18 participants were randomly chosenupfront. Overall, 54 experiments were conducted among 18participants.

A time limit of 15 minutes was provided to perform each exper-iment, and the participants were given the choice to continueor give up if it goes beyond 15 minutes. An instruction sheetwith detailed verbose instructions of the implementation taskwas provided in a written form before the start the task. Ademo video explaining the features of each view was played tothe participant just before performing the corresponding task.Additionally, users had access to internet throughout the ex-periment, and there were no restriction on referring to sourcesin the internet to complete the task. Throughout the task, oneinterviewer stayed in the room to help with infrastructural oroperational needs, however she did not indulge in any techni-cal discussion with the participant. All participants were giventhe same laptop and infrastructure to complete the task. Eachstudy lasted around 60 minutes and the participants were pro-vided with refreshments. To increase the competitive natureof the tasks, the top three participants were promised a rewardof 20 USD. The instructions sheets and videos, the qualitativesurvery results and the task-view combination of all the 18participants is available here https://dl-ide.github.io/.

At the end of each experiment, two different surveys wereconducted: (1) System Usability Scale (SUS) [14] to study

for usability of the corresponding view, and (2) NASA TaskLoad Index (TLX) [18] to study the experimental task load.Also, the time taken for the participant to complete a particularexperiment is noted along with the entire screen recording ofthe activities.

RESULTS OF USER STUDYThe results of study are analyzed and summarized in thissection.

Analysis of Accuracy and Time TakenEach of the 9 expert and 9 intermediate participants performedone experiment in each of the three views: code, tabular, andDL-IDE. We analzye the responses and results by answering5 research questions.

RQ1: “Does the visual drag-and-drop view enable users toimplement the deep learning models more accurately?". The

Figure 6: Plot showing the successful attempts by 9 expert and9 intermediate participants in each of the three views: code,tabular, and DL-IDE.

(a) All experiments (b) Only successful experiments

Figure 7: Plot showing the mean time duration by all users in doing in the experiments across the three different views: code,tabular, DL-IDE. The restricted time provided to the participants is 15 minutes (900 seconds) though they were later given a choiceto take additional infinite time to finish the task.

(a) Successful experiments

(b) Mean time duration

Figure 8: Plot showing the successful attempts and the meantime taken by 11 male and 7 female participants to performexperiments in three different views: code, tabular, DL-IDE.

(a) All experiments

(b) Only successful experiments

Figure 9: Plots showing the mean time duration taken by the18 participants in doing the tasks in experiment 1, 2, 3 in allthe three views.

Figure 10: Mean SUS score obtained for the three differentviews: code, tabular, and DL-IDE.

accuracy of the task was verified by executing the implementeddeep learning model from each of the three views.

As observed from Figure 6, 17 of the 18 participants were ableto finish the task successfully using the DL-IDE view, whileonly 4 participants were able to implement the task using thecode view (traditional programming) and only 8 participantswere successful with the tabular view. Note that this resultis despite providing infinite time beyond the instructed 15minutes to finish the experiment. Thus, it is safe to concludethat almost 78% (14/18) participants gave up without beingable to complete the deep learning model implementationusing coding. Out of these 18 participants, 11 coded usingKeras, 5 using Tensorflow, 2 used PyTorch. Irrespective of thechoice of the implementation library, it can be observed thatprogramming view remained a challenge.

Another reason for the higher success rate in DL-IDE can beattributed to the static validation component developed as apart of our system. Manually observing the screen record-ings of the experiments, we noted that more often participantsfound it highly challenging to debug the error in their imple-mented code and that majorly influenced them to give up. Thevalidation component provided easy guidance with precisemessage and highlighted specific layers for the users to fixthe bugs, improving the success rate for implementing deeplearning models in DL-IDE.

Also, preference for DL-IDE is supported by the qualitativesurvey conducted prior to the experiments where the partici-pants where asked the question, “When reading in research pa-pers, what style of deep learning model representation wouldyou prefer?" with choices as (a) Tabular rows and columns,or (b) Flow Figure as an Image, or (c) others. 15 of the 18participants preferred deep learning models being representedas a “Flow Figure" which is the representation in DL-IDE.

RQ2: “Can deep learning models be implemented much fasterusing the visual drag-and-drop view?".

Figure 7a depicts the mean and the standard deviation for timeduration, spent by participants in performing the experimentsacross different views. Figure 7b depicts the mean time takenand the standard deviation of only those experiments whichwere successfully completed by the participants. Irrespective

of the success outcome of the experiment, the mean time takenfor coding is much more than the visual methods of designingdeep learning models. The mean time taken for implementingdeep leaning models using either of visual programming viewis in the range of 8.9 − 11.8 minutes (536 − 705 seconds)while for the traditional programming view the mean timetaken is 27.4− 35 minutes (1646− 2098 seconds). Thoughthe mean time taken in tabular view is similar to DL-IDE, thesuccess rate of designing accurate deep learning designs in theDL-IDE view is much higher (more than 70%) as debuggingmodel designs is much more intuitive and natural.

RQ3: “Is DL-IDE suited well for both intermediate and expertusers in implementing deep learning models?".

From Figure 7a, it can be observed that the mean time durationto implement deep learning models for intermediate partici-pants (1646s) is much higher than for expert users (1245s), forcode view. However, using DL-IDE the difference in meantime duration is reduced to 705s for intermediate users and560s for expert users. It can also be observed that in DL-IDE,the variance of time taken is comparable irrespective of userexpertise, showing that DL-IDE caters to all the users equallyirrespective of their prior expertise in deep learning.

RQ4: “Does participant’s gender affect the outcome of theexperiments?".

Figure 8a and Figure 8b capture the successful attempts andthe mean time duration in performing the experiments acrossgender. The mean time spent by all the participants is similarin all the views. Also, gender does not act as a biasing factorin successful completion of the experiment as there is nosignificant difference across the gender.

RQ5: “Is there a learning bias displayed by the participantsbased on the sequence in which tasks are performed?".

Figure 9a and Figure 9b attempts to address an importantquestion of whether the sequence in which the views are pre-sented to the participant affect the mean time to complete theexperiment. Mean time across all the experiments is highlysimilar for DL-IDE and tabular views demonstrating that theorder does not affect the performance. However, it can beobserved in Figure 9a that the mean coding time is higher if itis presented as Experiment 1 compared to when it is presentedas Experiment 2 and Experiment 3. This can be attributedto the human tendency to spend more time in completing thetask for the initial experiments than for the later experiments,where there is a tendency to give up in lesser time. Figure 9bshows that even in successful experiments, the order does notaffect the time performance in DL-IDE and tabular views. Forcoding view, there were no successful attempts in Experiment2 and Experiment 3.

System Usability Scale (SUS)To study the usability of the three views, System UsabilityScale (SUS) [14] was used. It consists of a set of 10 questionswith every alternate question having a positive and negativeintent. Each question has five different choices ranging fromstrongly agree to strongly disagree. SUS is known to workwith smaller sample sizes, as well. A SUS score of 68 is con-

Figure 11: Plots describing the overall mean NASA TLX scoreas well as the NASA TLX score for each of the six dimensionsfor all the three views.

sidered average and any system with a SUS score of above 68is considered usable. The SUS questionnaire was conductedfor every participant at the end of every experiment. Figure 10indicates the mean SUS score computed for the different viewsbased on the user expertise. It can be observed that the codingview has an average SUS score of 60 for the intermediate par-ticipants and 68 for the experts based on their feedback. Thus,even for experts the usability of the system is just average.However, the average SUS score for DL-IDE is 90 and 89for intermediate and experts respectively, the highest in ourstudies. From our study, we can infer that the drag-and-dropinterface in DL-IDE can be considered as the highly usablesystem for both intermediate and experts, alike.

NASA TLX ResultsNASA Task Load Index (TLX) [18] is a popular tool usedto measure the perceived workload of a user after perform-ing a specific task. The questionnaire is multidimensional,measuring the workload impact across six different dimen-sions. The participants were provided with the NASA TLXThe plot in Figure 11 captures the overall workload faced bythe participants averaged along the six different dimensions.From the results obtained on the individual dimensions, weobserve that the mental demand such as looking, calculating,and searching required by the code view is significantly highas users often have to refer documentation or check syntax orperform layer size calculation. DL-IDE view has least demandfor mental strength as it provides easy to use interface forfundamental deep learning functionalities. The physical de-mands such as typing, scrolling, and clicking are less requiredin DL-IDE and tabular view as compared to code view. Boththe visual programming views have lesser physical demand,as they don’t involve much of typing and window switching.Both the visual programming interfaces require less temporaldemand, as the time to create or implement layers and connectthem in these views are less. Further, participants were notsatisfied with their performance in code view and tabular view.The key reason of high dissatisfaction is that they could notcomplete the experiments in given time which we observedfrom them as feedback post experiments. Effort required todesign a deep learning model in DL-IDE is less compared to

other views, as DL-IDE provides easy to use drag-and-dropinterface to create the deep learning models. The study alsoillustrates the frustration of the participants while performingthe experiments, where less effort and less demand of mentaland physical activity leads users to less frustration. The meanworkload follows the same footprint as other workload figuresand based on overall score we can conclude code view has thehighest workload than the tabular view or DL-IDE view.

DESIGN IMPLICATIONSThe results obtained in the user study are in favour of theproposed DL-IDE in terms of successful completion of tasks(94%), time required to perform a task (9−12 minutes), sys-tem usability (SUS score of 90), and least workload (NASATLX of 21). However, the obtained results may not be gener-alizable and the design study had a few implications, summa-rized as follows:

1. The user study is conducted using a set of 18 participantsfrom a restricted demography and age group.

2. The participants are students selected at random from var-ious universities. The prerequisites for a student to par-ticipate in the user study were proficiency in python andminimum development knowledge in deep learning. Stu-dents who had done a couple of projects in deep learningturned out to be experts and those who have done only oneor two projects turned out to be intermediate in our group ofparticipants. However, researchers in deep learning couldbe the actual set of experts and studies have to be conductedusing them to verify if similar results could be extrapolated.With the existing user studies, it is safe to conclude thata visual drag-and-drop based DL-IDE aids users in betterimplementing deep learning models.

3. The primary purpose of this research paper is to method-ically study and compare visual programming paradigm(drag-and-drop) vs. traditional programming paradigm indeep learning implementation. DL-IDE is one of the fewdrag-and-drop frameworks in the literature for deep learningimplementation, with enhanced features such as (1) NeuralLanguage Definition Standard (NLDS), (2) static designvalidation service, and (3) multiple library code genera-tion. Comparing different drag-and-drop frameworks forimplementing deep learning models is out of scope for thisresearch work and is considered as a future study.

On the positive side, the results obtained from the studiesconducted in this research has some strong, impacting impli-cations on the research and development in the deep learningcommunity. The major implications are summarized below:

1. Owing to the graphical structure of deep learning models, itis established that a visual drag-and-drop framework couldbe an efficient platform for implementing deep learningmodels compared to the traditional programming paradigm.This encourages the developers of libraries such as Keras,Tensorflow, PyTorch to align and support visual program-ming frameworks.

2. For students and beginners, the drag-and-drop frameworkprovides an easy, smooth, and confident route to adapt deep

learning with very low initial learning curve. Additionally,such visual tools could be used as effective teaching toolsin classrooms to intuitively explain the design choices in adeep learning model.

3. With the growing requirement of having deep learning asa skill for software engineers, industries are looking at up-skilling hundreds (may be thousands) of their existing soft-ware engineers and data scientists. DL-IDE could enablethis transition at industry level, democratizing deep learningimplementation for novice software engineers.

4. Existing implementation of research papers and algorithmsare mostly available in only a single library offering verylittle scope for interoperability into other libraries. With theaid of DL-IDE and abstract NLDS representation, standard-ization of deep learning models could be obtained. Researchpapers could make available their implementation in stan-dard NLDS format and different users could generate thesame implementation in different libraries of their choice.Furthermore, the flow figure used in research papers couldbe standardized as the visual drag-and-drop screenshot tomake research papers more uniform and easy to read.

CONCLUSION AND FUTURE WORKSIn the recent years, deep learning has been the driving force formost of the popular AI applications such as speech recognition,image processing, and natural language processing. Therefore,a lot of aspiring developers, students, and enterprises are want-ing to solve different problems using deep learning. However,the democratization of deep learning is not truly achieved andremains a tool for the experts because of the (1) high initiallearning curve, (2) lack of interaction across existing deeplearning libraries, and (3) need for theoretical deep learningknowledge.

To address these challenges, in this research work we presentedDL-IDE, an intuitive drag-and-drop based user interface forvisually constructing deep learning models. The tool providesan end-to-end development capability including model design-ing from scratch, model design validation, code generation inmultiple libraries, model training to deployment, and modelinference. Further, we validated the usability of our system byconducting comprehensive experiments with 18 participantson three different tasks of equal difficulty using three differentset of tools (views). The participants were able to success-fully complete the implementation task 94% of the times usingDL-IDE, however, only 22% of the times using the traditionalprogramming method with any library of their choice. On anaverage, participants took about 9−12 minutes to implementthe tasks using DL-IDE, while successful participants tookmore than 27 minutes to implement the tasks using program-ming. DL-IDE obtained a System Usability Scale (SUS) scoreof 90 compared to a score of 68 for programming, establishingthe ease and usability of DL-IDE. DL-IDE obtained the leastNASA-TLX workload index of 21 compared with a indexvalue of 55 for traditional programming.

As there exists multiple drag-and-drop frameworks in the liter-ature, it would be interesting to compare and study the multipleframeworks in terms of efficiency and usability. Also, user

studies could be conducted on a more diverse set of partici-pants and experts to obtain more generalizable conclusions.

REFERENCES1. 2018. Cloud-CV/Fabrik: Collaboratively build, visualize,

and design neural nets in browser.https://github.com/Cloud-CV/Fabrik. (2018). (Accessedon 05/11/2018).

2. 2018. DeepCognition - Become an AI-PoweredOrganization Today. http://deepcognition.ai/. (2018).(Accessed on 05/11/2018).

3. 2018. Home // AETROS - enterprise deep learningplatform, tools and services. https://aetros.com/. (2018).(Accessed on 05/11/2018).

4. 2018. Intel nGraph - Intel AI.https://ai.intel.com/intel-ngraph/. (2018). (Accessedon 05/15/2018).

5. 2018. lutzroeder/Netron: Visualizer for deep learning andmachine learning models.https://github.com/lutzroeder/Netron. (2018).(Accessed on 05/11/2018).

6. 2018. Machine UI. https://machineui.co/. (2018).(Accessed on 05/11/2018).

7. 2018a. Neural Network Console. https://dl.sony.com/.(2018). (Accessed on 05/11/2018).

8. 2018b. Neural Network Exchange Format. https://www.khronos.org/registry/NNEF/specs/1.0/nnef-1.0.pdf.(2018). (Accessed on 05/15/2018).

9. 2018. NVIDIA DIGITS | NVIDIA Developer.https://developer.nvidia.com/digits. (2018). (Accessedon 05/11/2018).

10. 2018. ONNX. https://onnx.ai/. (2018). (Accessed on05/15/2018).

11. 2018. TensorBoard: Visualizing Learning | TensorFlow.https://www.tensorflow.org/programmers_guide/

summaries_and_tensorboard. (2018). (Accessed on05/11/2018).

12. Anelia Angelova, Alex Krizhevsky, Vincent Vanhoucke,Abhijit S Ogale, and Dave Ferguson. 2015. Real-TimePedestrian Detection with Deep Network Cascades.. InBMVC, Vol. 2. 4.

13. Mariusz Bojarski, Davide Del Testa, DanielDworakowski, Bernhard Firner, Beat Flepp, PrasoonGoyal, Lawrence D Jackel, Mathew Monfort, Urs Muller,Jiakai Zhang, and others. 2016. End to end learning forself-driving cars. arXiv preprint arXiv:1604.07316(2016).

14. John Brooke and others. 1996. SUS-A quick and dirtyusability scale. Usability evaluation in industry 189, 194(1996), 4–7.

15. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L.Fei-Fei. 2009. ImageNet: A Large-Scale HierarchicalImage Database. (2009).

16. Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko,Susan M Swetter, Helen M Blau, and Sebastian Thrun.2017. Dermatologist-level classification of skin cancerwith deep neural networks. Nature 542, 7639 (2017), 115.

17. Eibe Frank, Mark A. Hall, and Ian H. Witten. 2016. TheWEKA Workbench. Online Appendix for "Data Mining:Practical Machine Learning Tools and Techniques",Morgan Kaufmann, Fourth Edition. (2016).

18. Sandra G Hart and Lowell E Staveland. 1988.Development of NASA-TLX (Task Load Index): Resultsof empirical and theoretical research. In Advances inpsychology. Vol. 52. Elsevier, 139–183.

19. Brody Huval, Tao Wang, Sameep Tandon, Jeff Kiske,Will Song, Joel Pazhayampallil, Mykhaylo Andriluka,Pranav Rajpurkar, Toki Migimatsu, Royce Cheng-Yue,and others. 2015. An empirical evaluation of deeplearning on highway driving. arXiv preprintarXiv:1504.01716 (2015).

20. Minsuk Kahng, Pierre Y. Andrews, Aditya Kalro, andDuen Horng Chau. 2017. ActiVis: Visual Exploration ofIndustry-Scale Deep Neural Network Models. CoRRabs/1704.01942 (2017). http://arxiv.org/abs/1704.01942

21. Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. 2014.CIFAR-10 (Canadian Institute for Advanced Research).(2014). http://www.cs.toronto.edu/~kriz/cifar.html

22. Fausto Milletari, Nassir Navab, and Seyed-AhmadAhmadi. 2016. V-net: Fully convolutional neuralnetworks for volumetric medical image segmentation. In3D Vision (3DV), 2016 Fourth International Conferenceon. IEEE, 565–571.

23. Brad A Myers. 1990. Taxonomies of visual programmingand program visualization. Journal of Visual Languages& Computing 1, 1 (1990), 97–123.

24. Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman,and others. 2015. Deep Face Recognition.. In BMVC,Vol. 1. 6.

25. Anush Sankaran, Rahul Aralikatte, Senthil Mani, ShreyaKhare, Naveen Panwar, and Neelamadhav Gantayat.2017. DARVIZ: Deep Abstract Representation,Visualization, and Verification of Deep Learning Models.In Proceedings of the 39th International Conference onSoftware Engineering: New Ideas and Emerging ResultsTrack (ICSE-NIER ’17). IEEE Press, Piscataway, NJ,USA, 47–50. DOI:http://dx.doi.org/10.1109/ICSE-NIER.2017.13

26. Terrence Joseph Sejnowski. 2018. The Deep LearningRevolution. MIT Press.

27. Yi Sun, Xiaogang Wang, and Xiaoou Tang. 2014. Deeplearning face representation from predicting 10,000classes. In Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition. 1891–1898.

28. Tianhao Zhang, Gregory Kahn, Sergey Levine, and PieterAbbeel. 2016. Learning deep control policies for

autonomous aerial vehicles with mpc-guided policysearch. In Robotics and Automation (ICRA), 2016 IEEEInternational Conference on. IEEE, 528–535.