JsConf 2014 - Google BigQuery API Node.js實作記錄
-
Upload
simon-su -
Category
Technology
-
view
449 -
download
3
description
Transcript of JsConf 2014 - Google BigQuery API Node.js實作記錄
![Page 1: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/1.jpg)
Simon @ MiCloud
![Page 2: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/2.jpg)
We are...
![Page 3: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/3.jpg)
前情提要
很久很久以前...
![Page 4: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/4.jpg)
時代不同,⾯面對資料的態度也不同...
● 存得起來的,叫做Storage (儲存)!
● 看得到的,叫做Data (資料)!
● 看得懂的,叫做Information (資訊)!
● ⽤用得出來的,才能夠叫做Intelligent (智慧)
和沛科技 CEO & Founder - 翟本喬
![Page 5: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/5.jpg)
資料...就是應該被儲存...● POS data
● Log data
● User behavior
● Data Warehouse
● Transactions
![Page 6: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/6.jpg)
貌似正常的⼈人
![Page 7: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/7.jpg)
看似無意義的log
![Page 8: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/8.jpg)
接下來,我們要開幹了!
![Page 9: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/9.jpg)
Tools
Req?
Web
App
Service
![Page 10: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/10.jpg)
絕世武功秘笈…
![Page 11: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/11.jpg)
![Page 12: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/12.jpg)
我們需要的是● ⼀一個存取的⽅方式 ● 越簡單越好...
● 輕量...
● 快速...
天下武功,唯快不破...
![Page 13: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/13.jpg)
Google BigQuery - 秘笈
![Page 14: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/14.jpg)
Node.js - 武功
select repository language from
372211 259329 223470
![Page 15: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/15.jpg)
![Page 16: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/16.jpg)
合體!!!
絕世武功秘笈
![Page 17: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/17.jpg)
But...⼈人⽣生最厲害就是這個BUT!
Hello Google! Login Required :)
WTF… =_=
![Page 18: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/18.jpg)
Node.js & Oauth & Google
Hello Google! What do u want =_=
Authenticate with Oauth2.0
![Page 19: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/19.jpg)
Tools
AuthReq?
Web
App
Service
![Page 20: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/20.jpg)
$ npm search oauth
![Page 21: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/21.jpg)
google-api-utility模組開發歷程
透過初始化設定之後, 即可以直接進⾏行api呼叫動作 !1. 設定檔抽離 2. 結合request模組進⾏行API呼叫
![Page 22: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/22.jpg)
google-api-utility module基本資訊 ● https://github.com/peihsinsu/google-api-utility 安裝 ● npm install google-api-utility 操作 ● apiutil.init(config) ● apiutil.request(options, callback)
![Page 23: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/23.jpg)
Google Oauth2Web Server Application Service Account
Request token
Authorization code
Exchange code for token
Token response
Use token to call Google API
Your App Google Servers
User login !& consent
User
Create and sign JWT
Use JWT to request token
Token response
Use token to call Google API
Server App
Google Servers
![Page 24: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/24.jpg)
Service Account
Configures Authenticate can be cache and auto refresh Others… Request… Request… Request…
Create and sign JWT
Use JWT to request token
Token response
Use token to call Google API
Server App
Google Servers
![Page 25: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/25.jpg)
免不了的⿇麻煩....
![Page 26: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/26.jpg)
Service Owner在Google的設定
![Page 27: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/27.jpg)
Prepare Authentications
![Page 28: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/28.jpg)
$ openssl pkcs12 -in privatekey.p12 -out privatekey.pem -nocerts
$ openssl rsa -in privatekey.pem -out key.pem
Generate key.pem
$ openssl pkcs12 -in privatekey.p12 -nodes -nocerts > key.pem
OR
![Page 29: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/29.jpg)
var auth = require('google-api-utility')
auth.init({
scope: 'https://www.googleapis.com/auth/bigquery https://
www.googleapis.com/auth/cloud-platform',
client_secret: '/path-to-client_secret.json',
key_pem: '/path-to-key.pem'
});
使⽤用範例 - 初始化
此處需要綁定所欲呼叫的API相關授權之Scope位置
設定client_secret.json與相關pem檔案位置,供jwt運算使⽤用
![Page 30: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/30.jpg)
使⽤用範例 - 呼叫BigQueryvar request = auth.request;
var bqurl = 'https://www.googleapis.com/bigquery/v2/projects/%s/datasets';request({ url: util.format(bqurl, project), method: 'GET' }, function(err, req, doc){ // implements });
同原request模組操作⽅方式
結合原request模組之function,供api呼叫使⽤用
![Page 31: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/31.jpg)
Tools
AuthReq?
APIWeb
App
Service
![Page 32: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/32.jpg)
Google API Explore HTTPS://developers.google.com/apis-explorer/
![Page 33: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/33.jpg)
Google API Explore - Query
![Page 34: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/34.jpg)
Google API Explore - Auth
Operation Scope
![Page 35: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/35.jpg)
Google API Explore - Response
![Page 36: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/36.jpg)
Tools
AuthReq?
APISDK Web
App
Service
![Page 37: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/37.jpg)
Idea...● bigquery.init({...configurations...})
● bigquery.[what].[do](...)
- bigquery.dataset.list(....)
- bigquery.table.load(..., callback)
![Page 38: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/38.jpg)
bigquery module基本資訊 ● https://github.com/peihsinsu/bigquery 安裝 ● npm install bigquery 操作 ● bigquery.init(config) ● bigquery.[category].[operation](options, callback)
![Page 39: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/39.jpg)
重新包裝 - bigquery模組var bq = require('bigquery') , prjId = 'your-bigquery-project-id';
bq.init({ client_secret: '/path/to/client_secret.json', key_pem: '/path-to-key.pem'});
bq.dataset.list(prjId, function(e,r,d){ if(e) console.log(e); console.log(JSON.stringify(d)); });
bigquery模組可參考:https://github.com/peihsinsu/bigquery
透過bq呼叫某個操作之下的function
![Page 40: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/40.jpg)
Source Code...var util = require('util') , auth = require('google-api-utility') , request = auth.request , _ = require('underscore') exports.init = auth.init; exports.job = { token: '', listds : function(project, cb){ var bqurl = 'https://www.googleapis.com/bigquery/v2/projects/%s/datasets'; request({ url: util.format(bqurl, project), method: 'GET' }, cb?cb:auth.commonCb); }, … (skip) }
封裝相同類別的api在⼀一起!ex: job相關的放在job物件中
![Page 41: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/41.jpg)
NOT ONLY Node.js Love JavaScript…..
![Page 42: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/42.jpg)
![Page 43: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/43.jpg)
The fancy integrate with Sheet
![Page 44: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/44.jpg)
More and more chart….
![Page 45: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/45.jpg)
Charting in Apps Script
![Page 46: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/46.jpg)
![Page 47: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/47.jpg)
Google provided node.js tool
![Page 48: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/48.jpg)
Operation with googleapisvar googleapis = require('googleapis'); !var jwt = new googleapis.auth.JWT( '[email protected]', '/path/to/key.pem', null, [
'https://www.googleapis.com/auth/bigquery', 'https://www.googleapis.com/auth/cloud-platform'
]);
![Page 49: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/49.jpg)
Operation with googleapisjwt.authorize(function(err, tokens) { googleapis.discover('bigquery', 'v2').execute(function(e,client) { if(e) console.log(e); else client.bigquery.datasets.list(param).withAuthClient(jwt)
.execute( function(err, response) {
if(err) console.log(err); console.log(JSON.stringify(response)); }); }); });
![Page 50: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/50.jpg)
Full Code
![Page 51: JsConf 2014 - Google BigQuery API Node.js實作記錄](https://reader031.fdocuments.net/reader031/viewer/2022013102/554a26ccb4c90526578b4a0f/html5/thumbnails/51.jpg)
HTTP://OpenNodes.arecord.us
OR
Sample
Copy
Run!