Serious SenchaSencha Data-Layer

Learned lessons from the first Sencha projects at ]project-open[frank.bergmann@project-open.com

Contents Architecture– Model– Store– Proxy

Practical Tricks– Performance

• “Dereferenced” columns• Compression• Store reuse

– Dependent stores– Error handling

Enterprise Reuse– Reusing models– Reusing stores– Sencha ExtJS vs. Touch reuse

Architecture

ServerServer

Architecture – The Role of Model, Store & Proxy

]po[ Server]po[ Server

RESTInterface

RESTInterface

Server:REST API

ClientClient

NoteNavigationViewNoteNavigationView

NoteListNoteList

NoteDetailNoteDetail

GUI View:Pages & Panels

NoteStoreNoteStore

• Id: 12345• note: „asdf@asdf.com“• note_type_id: 11508• note_status_id: 11400• object_id: 624

• Id: 12346• note: „http://www.test.com/“• note_type_id: 11510• note_status_id: 11400• object_id: 8868

. . .

NoteModelNoteModel• Id: integer• note: text• note_type_id: integer• note_status_id: integer• object_id: integer

„Proxy“„Proxy“

Data:Model & Store

Network:Interface

Model A “Model” defines an object type together with it’s attributes

Optionally attributes may have an object type defined:– String (default)– int (-eger)– boolean

Corresponds to a UML object definition or an SQL table definition.

Watch out for the “config” section. This is required for Sencha Touch not allowed in ExtJS

The model closely resembles the SQL data-model in ]project-open[. In fact, we can generate the model automatically from the ]po[ SQL metadata system.

Ext.define('PO.model.Note', { extend: 'Ext.data.Model', config: { fields: [ 'id', 'note', 'object_id', 'note_status_id', 'note_type_id' ] }});

NoteNote• Id: integer• note: text• note_type_id: integer• note_status_id: integer• object_id: integer

Store A “store” contains a list of models

A store corresponds to an SQL table with a list of models.

Stores are directly used by lists, widgets and other GUI elements for data.

The store specifies the “proxy” that defines how data are read/written from/to the server

Ext.define('PO.store.NoteStore', { extend: 'Ext.data.Store', storeId: 'noteStore', config: { model: 'PO.model.Note', autoLoad: true, proxy: { type: 'rest', url: '/intranet-rest/im_note', appendId: true, extraParams: { format: 'json' }, reader: { type: 'json', rootProperty: 'data' } } }});

Proxy A “proxy” defines how data are read/written from/to the server.

Reader: Specifications on how to convert JSON data from the server

Writer: Specifications on how to convert data into JSON for the server

Two options:– Adapt the Reader/Writer

to the WS interface of the server or

– Adapt the WS server interface to Sencha

– The ]po[ REST interface has been designed for Sencha compatibility

Ext.define('PO.store.NoteStore', { extend: 'Ext.data.Store', storeId: 'noteStore', config: { model: 'PO.model.Note', autoLoad: true, proxy: { type: 'rest', url: '/intranet-rest/im_note', appendId: true, extraParams: { format: 'json' }, reader: { type: 'json', rootProperty: 'data' } } }});

Server Side Interface

Options on how to build suitable data-sources for the server side

ServerServer

Server Side Interfaces

]po[ Server]po[ Server

RESTInterface

RESTInterface

Server:REST API

ClientClient

NoteNavigationViewNoteNavigationView

NoteListNoteList

NoteDetailNoteDetail

GUI View:Pages & Panels

NoteStoreNoteStore

• Id: 12345• note: „asdf@asdf.com“• note_type_id: 11508• note_status_id: 11400• object_id: 624

• Id: 12346• note: „http://www.test.com/“• note_type_id: 11510• note_status_id: 11400• object_id: 8868

. . .

NoteModelNoteModel• Id: integer• note: text• note_type_id: integer• note_status_id: integer• object_id: integer

„Proxy“„Proxy“

Data:Model & Store

Network:Interface

Server Side Interfaces: Protocols, Styles and Formatting

Protocol Options– HTTP 1.1– Web Sockets

Communication Styles– Plain AJAX– REST style

Payload Formatting Options– XML– JSON– CSV

]project-open[ has decided for REST style communication with JSON formatting:

– REST provides at least some type of standard– REST is “stateless”– Sencha ExtJS/Touch come with a REST “Proxy” that can both read and

write– The ]po[ REST interface has been designed to meet the Sencha protocol.

Server Side Interfaces: Example

Sencha request URL to Server

Answer from Server– “success”: true/false for

error handling– “message”: Error

message– “data”: The starting

point for the payload data

{"success": true, "message": "Data loaded", "data": [

{'date': '2014-06-16', 'value': 0.0},{'date': '2014-06-17', 'value': 43200.0},{'date': '2014-06-19', 'value': 43200.0},{'date': '2014-06-24', 'value': 187200.0},{'date': '2014-06-25', 'value': 187200.0},{'date': '2014-06-27', 'value': 187200.0}

]}

http://po40dev.project-open.net/intranet-reporting-dashboard/project-eva.json?diagram_project_id=168725&page=1&start=0&limit=25

Server Side Interfaces: Store Example

The source code at the right is a real-world example.

“fields”: Defines the “columns” of the store

“proxy”: Defines how to load the data

“extraParams”: Allows to specify URL parameters – in this case we pass the project_id.

“reader”: Defines how to parse the response from the server.

projectEvaStore = Ext.create('Ext.data.Store', {fields: ['date', 'value'],autoLoad: true,proxy: {

type: 'rest',url: '/intranet-dashboard/project-eva.json',extraParams: {

project_id: project_id},reader: { type: 'json', root: 'data' }

}});

{"success": true, "message": "Data loaded", "data": [

{'date': '2014-06-16', 'value': 0.0},{'date': '2014-06-17', 'value': 43200.0},{'date': '2014-06-19', 'value': 43200.0},{'date': '2014-06-24', 'value': 187200.0},{'date': '2014-06-25', 'value': 187200.0},{'date': '2014-06-27', 'value': 187200.0}

]}

http://po40dev.project-open.net/intranet-reporting-dashboard/project-eva.json?project_id=168725&page=1&start=0&limit=25

Server Side Interfaces: Data Sources Considerations

A “data-source” is a server-side script that returns JSON data to the client

Security: Data-sources provide data to the “wild” Internet and are vulnerable to SQL injection, DoS, XSS and other attacks.

Permissions: The system’s permission model needs to honored

Performance: SQL optimization and caching

You may very quickly run into a variety of badly documented data-sources with security holes.

Plan for your future data-sources and keep order

Server Side Interfaces: Data Sources Examples

In ]po[ we use three different types of data-sources:– SQL as data-source:

A library allows to generically deploy any SQL command as a data-source including permissions

– ]po[ REST Interface:A generic REST interface that provides generic read, write, create and delete access to ]po[ business objects via the ]po[ SQL metadata system.

– Custom written in TCL (could be PHP): Only in special cases of complex logic or performance issues. Try to avoid whenever possible.

Practical Tricks

The Problem– An app is supposed to show the list of

projects for the current user. – Project information is stored in the

“Project” model that closely resembles the ]po[ “im_projects” table.

– The table stores the project’s customer and project manager as IDs, based on SQL design best practices (1st normal form).

– => The app can not show the project list, unless it has loaded the full Users and Customers stores, which can take minutes (with 40.000 customers, for example).

The Solution– “Dereferencing”:

The server-side REST interface includes the user-name and customer-name, together with the IDs

– Compression:Long lists of objects can be compressed on the server side using GZip (HTTP standard). This reduces the data size by a factor of 10.

– Client-side caching:Lists of slow changing data (users, customer, value ranges (categories), …) can be stored in the browser. These data can then be “synced” instead of being loaded.

Performance issues with large amounts of data

ProjectProject• project_id: integer• […] • customer_id: integer• project_manager_id: integer• […]

CustomerCustomer• customer_id: integer• customer_name: string• […]

UserUser• user_id: integer• user_name: string• […]

Dependent Stores – Stores with similar contents

The Problem– The client has to load multiple times

similar data from the server.– Reason: Every list, drop-down box, etc.

GUI element requires a store for it’s data.– These stores frequently contain similar

data, but not exactly the same data:• Different filters or subsets (Users of group

“Employees” vs. “Customers”)• Different sort order• Different groupings

– These stores sometimes are long.

The Solution– Load the maximum list in the background

right after the initial startup– After loading the store, create the

“dependent” stores using a JavaScript procedure.

Error Handling The Problem:– Many things can go wrong on the server-side

• Connection unavailable• Server unavailable• Database issues• Unique and check condition violation• Software error• …

– These issues are somehow “obscure” and difficult to debug:• They are part of a long chain of processes• They may occur rarely or randomly• They may occur under high load or concurrency conditions

– The issues may be difficult to communicate• They appear to technically unskilled users.• They appear on a mobile device with limited GUI capabilities

– When operating the service, you will need to be able to debug these issues efficiently

The Solution– All parts of the chain need to return reasonable error

messages.– Display important errors to the user– Provide a server-side error reporting functionality where the

user can submit error reports.– Keep an error log on the client side. You may send this error

log to the server together with a stack trace in order to provide a context for debugging.

– ]project-open[ includes a package for submitting and tracking error reports per user and software version

Enterprise Reuse

Enterprise Reuse Motivation

Enterprise applications consist of hundreds or thousands of pages

The page structure is relatively simple and standardized, as opposed to sophisticated user experiences of apps mainly created for marketing reasons.

Reuse becomes more important because of the number of pages

Reuse is possible, because the various pages are more standard.

Enterprise Reuse –REST Interface

Generic REST interface for business objects based on SQL metadata– Generic R=read operation returns table

columns plus optionally dereferenced object_ids

– Generic W=write operation updates existing objects

– Generic C=create operation is not possible in ]project-open[, but may be possible in other business applications because of object type specific PL/SQL constructors

– Generic D=delete operation not possible in ]po[

Permissions– Permissions per operation and per

attribute need to be handled.– Access to certain business objects may

need to be logged for security reasons

Enterprise Reuse –Automatic Models

Models can be generated automatically based on SQL metadata (table columns, column types and foreign key constraints).

ThankYou!

frank.bergmann@project-open.comwww.project-open.com

(Frank Bergmann and Klaus Hofeditz)