GCP TensorFlow

Google Cloud ML Engine & Dataflow

H.Yoshikawa (@hayatoy)TFUG #524 May 2017

@hayatoy

APAC

TFUGGCPUGGDG

Presenter

Google Cloud ML Engine

Jupyter Notebook [DEMO]

Online Prediction [DEMO]

Google Cloud Dataflow

Cloud ML Engine

Cloud ML Engine

TensorFlow

Online Prediction

←ktkr!

Cloud ML Engine

Cloud ML Engine

Package

 gcloud 

REST APIPackage

Jupyter Notebook ←New!

 gcloud GCP

gcloud ml-engine jobs submit

COMMAND [GLOBAL-FLAG …]

REST API

Python Client Library

from oauth2client.client import GoogleCredentialsfrom googleapiclient import discovery

credentials = GoogleCredentials.get_application_default()ml = discovery.build('ml', 'v1', credentials=credentials)

job_req = ml.projects().jobs().create( parent=project_id, body=param)job_req.execute()

Jupyter Notebook

Notebook Cloud ML Engine

https://github.com/hayatoy/cloudml‑magic

Online Prediction ‑ Demo

pix2pix GCPUG

bit.ly/realgcpug

pix2pix

GCPUG

Original Edge 20 epochs 200 epochs

Online Prediction

SavedModel v1

serving_input_fn

Beta

Cloud Dataflow

Cloud Dataflow

ETL

tf.Transform

Dynamic Work Rebalancing

GCE

MapReduce

Dataflow

Dataflow

BigQuery, GCS

query = "SELECT * FROM [project:dataset.table]"

p = beam.Pipeline(options=options)

(p | 'query image list' >> beam.Read(beam.io.BigQuerySource(query=query))

| 'download image' >> beam.FlatMap(download_image)

| 'crop and resize image' >> beam.Map(crop_image)

| 'make edge image' >> beam.Map(make_edge_image)

1000

DWR

Cap opt.max_num_workers = 10 

Dataflow

GCP

Thank you!

Cloud ML Engine

https://cloud.google.com/ml‑engine/

Cloud Dataflow

https://cloud.google.com/dataflow/

Cloud ML Engine ‑ Jupyter Notebook Extension

https://github.com/hayatoy/cloudml‑magic