Nodejs.meetup

NYC Data Science Academy

Node.JS in Action

(Copyright materials, all right reserved)

Sign up for Node.js beginner class at www.nycdatascience.com

or email [email protected]


Overview

Deployment of Node.js on AWS EC2 Instance

Use Bootstrap Framework for our page

Rules of Routes, Session and MongoDB, Error Handling and Visiting Control

Web Scrapper

Save JSON data into MongoDB


AWS Security Group

You should Add HTTP Rules

For convenience, you could allow All TCP rules


Installation on Ubuntu

Our machine is Ubuntu 12.04 LTS, 64 bits

We install node and express from apt-get

Check if we are having the right configuration

$ sudo add-apt-repository ppa:chris-lea/node.js

$ sudo apt-get update

$ sudo apt-get install nodejs

$ sudo npm install express@3 -g

$ sudo apt-get install unzip

$ sudo apt-get install build-essential

$ node -v

v0.10.26

$ npm -v

1.4.3

$ express -V

3.5.1


Start Our first Project

In ~/ directory, we can start our first express project

If you see the notification, then it is OK!

Visit your public address on port 3000 and you will see

$ express -e nodejs-demo

$ cd nodejs-demo

$ sudo npm install

$ node app.js


Supervisor

If we modify our code, we need to restart our server. That is inconvenient.

By supervisor, we can modify our code and the server will restart automatically.

$ sudo npm install supervisor -g

$ supervisor app.js

Running node-supervisor with

program 'app.js'

--watch '.'

--extensions 'node,js'

--exec 'node'

Starting child process with 'node app.js'

Watching directory '/home/ubuntu/nodejs-demo' for changes.

Express server listening on port 3000


Intro to the working directory

node_modules: Store all the dependent libraries(every project manage its own

dependencies).

package.json: Project dependencies configuration and developer information

app.js: Application starting file

public: Static files(css,js,img)

routes: Routes files(C in MVC, controller)

Views: Page files(Ejs template)


Support html files

And rename views/index.ejs to views/index.html

Add the following lines in app.js, see next page for codes

Visit your website

$ mv views/index.ejs views/index.html


Support html files

var express = require('express');

var routes = require('./routes');

var user = require('./routes/user');

var http = require('http');

var path = require('path');

var ejs = require('ejs');

var app = express();

// all environments

app.set('port', process.env.PORT || 3000);

app.set('views', path.join(__dirname, 'views'));

// app.set('view engine', ‟ejs');

app.engine('.html', ejs.__express);

app.set('view engine', 'html');

app.use(express.favicon());

app.use(express.logger('dev'));


Use Bootstrap Framework

Bootstrap Framework is one of the most popular frameworks for front-end.

Download it from http://getbootstrap.com/ and unzip.

And we also need jQuery.

Check whether your “~/nodejs-demo/public/stylesheets” has three .css files and

whether „~/nodejs-demo/public/javascripts‟ has two .js files.

$ cd ~/

$ wget https://github.com/twbs/bootstrap/releases/download/v3.1.1/bootstrap-

3.1.1-dist.zip

$ unzip bootstrap-3.1.1-dist.zip

$ cd ~/bootstrap-3.1.1-dist

$ cp css/*.min.css ~/nodejs-demo/public/stylesheets/

$ cd js/

$ wget http://code.jquery.com/jquery-1.9.1.min.js

$ cp *.min.js ~/nodejs-demo/public/javascripts

$ cd ~/nodejs-demo

$ ls public/javascripts

$ ls public/stylesheets

http://getbootstrap.com/


Split index.html

We want a common header and footer by spliting index.html into header.html,

index.html and footer.html

First, we add these two news files in foldr views/

header.html

We edit the header.html like this

<!DOCTYPE html>

<html lang="en">

<head>

<meta charset="utf-8">

<title><%=: title %></title>



<link href="/stylesheets/bootstrap.min.css" rel="stylesheet" media="screen">

<!-- <link href="css/bootstrap-responsive.min.css" rel="stylesheet" media="screen"> --

>

</head>

<body screen_capture_injected="true">

$ vi ~/nodejs-demo/views/header.html


footer.html

We edit the footer.html like this

<script src="/javascripts/jquery-1.9.1.min.js">

</script>

<script src="/javascripts/bootstrap.min.js">

</script>

</body>

</html>

$ vi ~/nodejs-demo/views/footer.html


index.html

Through ejs(emmeded javascript), we can include our header and footer in

index.html

modify index.html

<% include header.html %>

<h1><%= title %>

</h1>

<p>Welcome to <%= title %>

</p>

<% include footer.html %>

$ rm ~/nodejs-demo/views/index.html

$ vi ~/nodejs-demo/views/index.html


Routes of our demo website

Routes: pages, authorization and action

/ : index.html, No need to login to visit.

/home : home.html, Need to login to visit.

/login : login.html, redirect to home.html with correct username and password

/logout : No corresponding html file

redirect back to index.html after logout

We can add routes to enable these features.


Add routes

In app.js, we edit it by “vi ~/nodejs-demo/app.js”

Note: app.get is get request, app.post is post request, get.all is request for this

path.

app.get('/', routes.index);

//app.get('/users', user.list);

app.get('/login', routes.login);

app.post('/login', routes.doLogin);

app.get('/logout', routes.logout);

app.get('/home', routes.home);


Add routes

Then we edit routes/index.js by by “vi ~/nodejs-demo/routes/index.js”


Add routes

exports.index = function(req, res){

res.render('index', { title: 'Express' });

};

exports.login = function(req, res){

res.render('login', { title: 'User Login'});

};

exports.doLogin = function(req, res){

var user={

username:'admin',

password:'admin'

}

if(req.body.username===user.username && req.body.password===user.password){

res.redirect('/home');

}

res.redirect('/login');

};

exports.logout = function(req, res){

res.redirect('/');

};

exports.home = function(req, res){

var user={

username:'admin',

password:'admin'

}

res.render('home', { title: 'Home',user: user});

};


Add pages

Create new home page by “vi ~/nodejs-demo/views/home.html” , see page 20

Create new login page by “vi ~/nodejs-demo/views/login.html”, see page 21


Create home.html


<h1>Welcome <%= user.username %>!</h1>

<a class="btn" href="/logout">Logout</a>



Create login.html


<link href="/stylesheets/signin.css" rel="stylesheet">

<div class="container">

<form class="form-signin" method="post">

<h2 class="form-signin-heading">Please sign in</h2>

<input type="text" class="form-control" id="username" name="username"

required autofocus>

<input type="password" class="form-control" id="password"

name="password" required>

<button class="btn btn-lg btn-primary btn-block" type="submit">Sign

in</button>

</form>

</div>



Add pages

And we edit index.html by „vi ~/nodejs-demo/views/index.html‟


<h1><%= title %>

</h1>

<p>Welcome to <%= title %>

<a href="/login">Signin</a>

</p>



Improve the look

This is what we see at /login.html:

Is there anyway we can make it better?


Edit the Bootstrap Style

We create signin.css file to public/stylesheets/ by

“vi ~/nodejs-demo/public/stylesheets/signin.css”

Copy/past left side and right side into the file

body {

padding-top: 40px;

padding-bottom: 40px;

background-color: #eee;

}

.form-signin {

max-width: 330px;

padding: 15px;

margin: 0 auto;

}

.form-signin .form-signin-heading,

.form-signin .checkbox {

margin-bottom: 10px;

}

.form-signin .checkbox {

font-weight: normal;

}

.form-signin .form-control {

position: relative;

font-size: 16px;

height: auto;

padding: 10px;

-webkit-box-sizing: border-box;

-moz-box-sizing: border-box;

box-sizing: border-box;

}

.form-signin .form-control:focus {

z-index: 2;

}

.form-signin input[type="text"] {

margin-bottom: -1px;

border-bottom-left-radius: 0;

border-bottom-right-radius: 0;

}

.form-signin input[type="password"] {

margin-bottom: 10px;

border-top-left-radius: 0;

border-top-right-radius: 0;

}


Edit the Bootstrap Style

Then the website looks like:

Use admin and admin to login!


Session

In “routes/index.js” file , when we call function “exports.doLogin”, if username and

password are correct, we will redirect to home.

In the same file, when we call function “exports.home”, we render the page and pass

username to the home.html by the following specific:

Why can't we assign the username to session so that we do not need pass to each

page again?

res.redirect('/home');

res.render('home', { title: 'Home',user: user});


Storage of data

On local drive, we store user credentials into cookie files.

On server side, we also store session and user credentials in database, like Redis,

MongoDB.

MongoDB pass the username object to each page instead of asking node.js passing

the username object between pages.


Edit app.js

The Order of Commands Matters !!!


Edit app.js

app.set('view engine', 'html');

app.use(express.favicon());

app.use(express.logger('dev'));

app.use(express.bodyParser());

app.use(express.json());

app.use(express.urlencoded());

app.use(express.methodOverride());

app.use(express.cookieParser());

app.use(express.cookieSession({secret :

'nycdatascience.com'}));

app.use(express.session({

secret : 'nycdatascience.com',

store: store,

cookie: { maxAge: 900000 }

}));

app.use(function(req, res, next){

res.locals.user = req.session.user;

next();

});

app.use(app.router);

app.use(express.static(path.join(__dirname, 'public')));

var express = require('express');

var routes = require('./routes');

var user = require('./routes/user');


var path = require('path');

var ejs = require('ejs');

var SessionStore = require("session-

mongoose")(express);

var store = new SessionStore({

url: "mongodb://localhost/session",

interval: 120000

});

var app = express();


MongoDB on Ubuntu

Following the official document, we successfully installed MongoDB

http://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/

$ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10

$ echo 'deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen' |

sudo tee /etc/apt/sources.list.d/mongodb.list

$ sudo apt-get update

$ sudo apt-get install mongodb-org

$ sudo service mongod start









Installing Mongoose

We need to install Mongoose: how node.js connect with MongoDB

Then /login page is accessible

$ sudo npm install session-mongoose

$ npm install mongoose


Edit index.js

We need to edit index.js to support session by „vi ~/nodejs-demo/routes/index.js‟


Edit index.js


var user={

username:'admin',

password:'admin'

}


req.session.user=user;

return res.redirect('/home');

}

else{

return res.redirect('/login');

}

};

exports.logout = function(req, res){

req.session.user=null;

res.redirect('/');

};

exports.home = function(req, res){

//delete first 5 lines, change last line

res.render('home', { title: 'Home'});

}


Error Handling

We want to tell user if they have entered the wrong information

Edit app.js by „vi ~/nodejs-demo/app.js‟

app.use(function(req, res, next){

res.locals.user = req.session.user;

var err = req.session.error;

delete req.session.error;

res.locals.message = '';

if (err) res.locals.message = '<div class="alert alert-error">' + err +'</div>';

next();

});


app.use(express.static(path.join(__dirname, 'public')));


Error Handling

Edit login.html by „vi ~/nodejs-demo/views/login.html‟


<link href="/stylesheets/signin.css" rel="stylesheet">

<div class="container">

<form class="form-signin" method="post">

<%- message %>

<h2 class="form-signin-heading">Please sign in</h2>

<input type="text" class="form-control" id="username" name="username" required autofocus>

<input type="password" class="form-control" id="password" name="password" required>

<button class="btn btn-lg btn-primary btn-block" type="submit">Sign in</button>

</form>

</div>



Error Handling

Edit index.js by by „vi ~/nodejs-demo/routes/index.js‟


var user={

username:'admin',

password:'admin'

}


req.session.user=user;

return res.redirect('/home');

} else {

req.session.error='Wrong Username or Password!';


}

};


Error Handling

Let's have a try:


Website safety

Our website is almost ready, but it is not safe yet.

We can access /home without login.


Refine model of our demo website

/: anybody could visit

Still remember app.get, app.post and app.all?

/login: use app.all to take care of the requests of visiting /login. We call

notAuthentication function(If session.user is not null, then the user has logined in.)

first to check whether user's been logged in

/logout: use app.get to take care of the requests of visiting /logout. We call

authentication function(If session.user is null, it will hint “please sign in”).

/home: use app.get to take care of the requests of visiting /home. We call

authentication function(If session.user is null, it will hint “please sign in”).


Edit app.js

We add two functions and make use of them.


Edit app.js


app.use(express.static(path.join(__dirname,

'public')));

function authentication(req, res, next) {

if (!req.session.user) {

req.session.error='Please Sign In to

Visit';


}

next();

}

function notAuthentication(req, res, next) {

if (req.session.user) {

req.session.error='Logged in';

return res.redirect(‟/home‟);

}

next();

}

// development only

if ('development' == app.get('env')) {

app.use(express.errorHandler());

}

app.get('/', routes.index);

//app.get('/users', user.list);

app.all('/login', notAuthentication);

app.get('/login', routes.login);

app.post('/login', routes.doLogin);

app.get('/logout', authentication);

app.get('/logout', routes.logout);

app.get('/home', authentication);

app.get('/home', routes.home);


Safety check

Let's check if it works.

We have done it!

Go to /home and /logout page while you are not logined in

Go to login page while you are logined in


Add some content?

So, how to add content to our database?

Why not use informations from other websites?


Install libraries

We need to install jsdom, jQuery, xmlhttprequest, request, htmlparser

$ sudo npm install jsdom

$ sudo npm install jQuery

$ sudo npm install xmlhttprequest

$ sudo npm install request

$ sudo npm install htmlparser


What is valuable?

Our event webpage!

http://www.meetup.com/NYC-Open-Data/events/163300552/







Add codes

Add myUtil.js by „vi ~/nodejs_demo/myUtil.js‟

It is used as getter function to scrap certain web page.

var MyUtil = function () {

};


var request = require('request');

MyUtil.prototype.get=function(url,callback){

request(url, function (error, response, body) {

if (!error && response.statusCode == 200) {

callback(body,response.statusCode);

}

})

}

module.exports = new MyUtil();


Add codes

And edit index.js

Add the following codes

in the end.

var myUtil = require('../myUtil.js');

var $ = require('jQuery');


var url="http://www.meetup.com/NYC-Open-Data/events/163300552/";

console.log(url);

myUtil.get(url,function(content,status){

console.log("status:="+status);

res.send(content);

});

};


Content Downloaded

Then we visit the index page:


Extract useful information

We keep modify routes/index.js and use XPath command to extract event name and

event start time:

Refresh the page, http://ec2-54-86-42-76.compute-1.amazonaws.com:3000/

In the console, we can see that the name and time are correct


Extract useful information





console.log(url);



var events={}

events.name = $(content).find('h1[itemprop="name"]').text();

events.time = $(content).find('time[id="event-start-time"]').text();

console.log(events);

res.send(content);

});

};


Use MongoDB to save JSON data

We first create a unique folder for our database operations

Then we create a file models/Event.js:

$ mkdir models



Then we add the model for events: Event.js

var mongoose = require('mongoose')

mongoose.connect('mongodb://localhost/nodejs');

exports.mongoose = mongoose;

var Schema = mongoose.Schema;

var EventSchema = new Schema({

name : String,

time : String,

});

var Event = mongoose.model("Event", EventSchema);

var EventExport = function(){};

module.exports = new EventExport();

EventExport.prototype.save = function(obj, callback) {

var instance = new Event(obj);

instance.save(function(err){

callback(err);

});

};



Finally, we edit index.js to call the save method:



Refresh our page, the content on the page and console is still the same.How about

the database?

We use

to interact with MongoDB

Then type the following command:

That is our data!

$ mongo

> use nodejs

switched to db nodejs

> show collections

events

system.indexes

> db.events.find()

{ "_id" : ObjectId("535526ee43059424331b16a9"), "name" : "Node.js Workshop II:

toolkit to make your work efficiently", "time" : "Monday, April 21, 2014 7:00 PM",

"__v" : 0 }


FAQ

Nodejs.meetup

Education

Transcript of Nodejs.meetup