Откуда движется волна? EMC_ECS_v2.pdfи требований ... – Intel Dual...
Transcript of Откуда движется волна? EMC_ECS_v2.pdfи требований ... – Intel Dual...
![Page 2: Откуда движется волна? EMC_ECS_v2.pdfи требований ... – Intel Dual Quad Core x86 Nodes • Dual 4-core IvyBridge (gen1) • Dual 6-core Haswell (gen](https://reader033.fdocuments.net/reader033/viewer/2022050212/5f5e9d1dae68d0169d3b5d86/html5/thumbnails/2.jpg)
© Dell Inc., 2017 г.2
Откуда движется волна?• Новые источники данных
Исследования в нефте-газовом секторе
Медицинские изображения
ВидеонаблюдениеМедиа
Умные датчики
WEB 3.0Internet of Things
Темные данные
![Page 3: Откуда движется волна? EMC_ECS_v2.pdfи требований ... – Intel Dual Quad Core x86 Nodes • Dual 4-core IvyBridge (gen1) • Dual 6-core Haswell (gen](https://reader033.fdocuments.net/reader033/viewer/2022050212/5f5e9d1dae68d0169d3b5d86/html5/thumbnails/3.jpg)
© Dell Inc., 2017 г.3
Две платформы
Традиционные системы
Стабильные и надёжные платформы для традиционных
задач
Новый тип платформ
Гибкая платформа для мгновенных изменений задач
и требований
![Page 4: Откуда движется волна? EMC_ECS_v2.pdfи требований ... – Intel Dual Quad Core x86 Nodes • Dual 4-core IvyBridge (gen1) • Dual 6-core Haswell (gen](https://reader033.fdocuments.net/reader033/viewer/2022050212/5f5e9d1dae68d0169d3b5d86/html5/thumbnails/4.jpg)
© Dell Inc., 2017 г.4
© Copyright 2017 Dell Inc.4
ISILON + ECS = от 20ТБ до ∞
УНИКАЛЬНОЕ РЕШЕНИЕ
для работы с неструктурированными данными
![Page 5: Откуда движется волна? EMC_ECS_v2.pdfи требований ... – Intel Dual Quad Core x86 Nodes • Dual 4-core IvyBridge (gen1) • Dual 6-core Haswell (gen](https://reader033.fdocuments.net/reader033/viewer/2022050212/5f5e9d1dae68d0169d3b5d86/html5/thumbnails/5.jpg)
© Dell Inc., 2017 г.5
Cloud Native Apps
(web/mobile)Cloud Backup Sync
& ShareAnalyticsIoTTiered
Archive
Cloud
Gateway
SITE 1SITE 2
SITE 3
Traditional/“Platform 2” Cloud Native/“Platform 3”
Scale Effortlessly - Store Efficiently - Access Globally
ECS – “Облачная” СХД из коробки
![Page 6: Откуда движется волна? EMC_ECS_v2.pdfи требований ... – Intel Dual Quad Core x86 Nodes • Dual 4-core IvyBridge (gen1) • Dual 6-core Haswell (gen](https://reader033.fdocuments.net/reader033/viewer/2022050212/5f5e9d1dae68d0169d3b5d86/html5/thumbnails/6.jpg)
© Dell Inc., 2017 г.6
Решение архивных задач на уровне Geo-scale
ECS в ногу со временем!!!
• Modern Hyper-scale Cloud Architecture– Scales from Petabytes to Exabytes
• Archive data from all sources– S3, Swift, Atmos and Centera CAS Object APIs
– HDFS compatible with Cloudera, Hortonworks, Pivotal etc.
• Break down barriers with Geo-scale data access
• Innovation to enable scalability, efficiency and serviceability!
ATMOS
![Page 7: Откуда движется волна? EMC_ECS_v2.pdfи требований ... – Intel Dual Quad Core x86 Nodes • Dual 4-core IvyBridge (gen1) • Dual 6-core Haswell (gen](https://reader033.fdocuments.net/reader033/viewer/2022050212/5f5e9d1dae68d0169d3b5d86/html5/thumbnails/7.jpg)
© Dell Inc., 2017 г.7
EMC Elastic Cloud Storage (ECS)
U-Series
Available in multiple capacities within a rack
x86 servers, JBOD DAS, 10GbE connectivity,
SATA/SAS Disks
Max 60 disks per DAE per node
D-Series
Denser Model
Minimum of eight x86 servers
Max 98 disks per DAE per node.
Hyper-scale
Aggressive seeking lowest $/GB
![Page 8: Откуда движется волна? EMC_ECS_v2.pdfи требований ... – Intel Dual Quad Core x86 Nodes • Dual 4-core IvyBridge (gen1) • Dual 6-core Haswell (gen](https://reader033.fdocuments.net/reader033/viewer/2022050212/5f5e9d1dae68d0169d3b5d86/html5/thumbnails/8.jpg)
© Dell Inc., 2017 г.8
ECS Appliance – Аппаратные составляющие
Data Network - Dual 10 GBE– Arista Switches
– Each ECS node connects to these switches via
dual 10GigE
– Switches connect to customer network (up to 8
uplinks per switch)
Internal Management Network – Single Gigabit Switch
– Optional uplink to customer network for
network based server management
Servers (ECS nodes)– Intel Dual Quad Core x86 Nodes
• Dual 4-core IvyBridge (gen1)
• Dual 6-core Haswell (gen 2)
– 64GB Memory
– 4 nodes in 2U chassis
– OS, HAL, Fabric & ViPR Data Services
software package
– Each node connects to a Disk Enclosure via
SAS
Disks & Enclosures– 6TB SATA (gen 1) or 8TB SAS Disks (Gen2,
D-Series supports both 8TB and 10 TB )
– Up to 60 or 98 Disks / 4U Enclosure
1
2
3
4
1
2
3
4
![Page 9: Откуда движется волна? EMC_ECS_v2.pdfи требований ... – Intel Dual Quad Core x86 Nodes • Dual 4-core IvyBridge (gen1) • Dual 6-core Haswell (gen](https://reader033.fdocuments.net/reader033/viewer/2022050212/5f5e9d1dae68d0169d3b5d86/html5/thumbnails/9.jpg)
© Dell Inc., 2017 г.9
ECS Software на серверах x86
EM
CC
usto
mers
Dell R730xd 13G
Minimum 5 servers
Dell DSS7000
![Page 10: Откуда движется волна? EMC_ECS_v2.pdfи требований ... – Intel Dual Quad Core x86 Nodes • Dual 4-core IvyBridge (gen1) • Dual 6-core Haswell (gen](https://reader033.fdocuments.net/reader033/viewer/2022050212/5f5e9d1dae68d0169d3b5d86/html5/thumbnails/10.jpg)
© Dell Inc., 2017 г.10
Объектный доступ
Namespace / Bucket|Container
• Namespace can span multiple instances of physical hardware, and data
management functions like data replication and data distribution at object-level
granularityy.
• Instead of organizing files in a directory hierarchy, object storage systems store files
in a flat organization of containers/buckets.
![Page 11: Откуда движется волна? EMC_ECS_v2.pdfи требований ... – Intel Dual Quad Core x86 Nodes • Dual 4-core IvyBridge (gen1) • Dual 6-core Haswell (gen](https://reader033.fdocuments.net/reader033/viewer/2022050212/5f5e9d1dae68d0169d3b5d86/html5/thumbnails/11.jpg)
© Dell Inc., 2017 г.11
CUSTOMER META-DATASYSTEM META-DATA
ECS – Как выглядят данные в объектной системе
DATA• Identifiers and
descriptors
• Encryption keys in
encrypted format
• Internal flags
• Location information
• Timestamps
• Configuration/tenancy
information
• Client=DellEMC
• Event=DellEMC Forum
• ID=123
![Page 12: Откуда движется волна? EMC_ECS_v2.pdfи требований ... – Intel Dual Quad Core x86 Nodes • Dual 4-core IvyBridge (gen1) • Dual 6-core Haswell (gen](https://reader033.fdocuments.net/reader033/viewer/2022050212/5f5e9d1dae68d0169d3b5d86/html5/thumbnails/12.jpg)
© Dell Inc., 2017 г.12
Протоколы доступа
• REST:
–Comibnation of HTTP methods
–Amazon S3
–Openstack Swift
–Atmos
–Content Adress Storage (CAS)
![Page 13: Откуда движется волна? EMC_ECS_v2.pdfи требований ... – Intel Dual Quad Core x86 Nodes • Dual 4-core IvyBridge (gen1) • Dual 6-core Haswell (gen](https://reader033.fdocuments.net/reader033/viewer/2022050212/5f5e9d1dae68d0169d3b5d86/html5/thumbnails/13.jpg)
© Dell Inc., 2017 г.13
import fileinput
from boto.s3.connection import S3Connection, OrdinaryCallingFormat, Key
# Считываем параметры доступа из файла, строки добвляем в список
config_file='C:/Users/vladim/Desktop/asd.txt'
config_file_list=[]
with open(config_file) as f:
for line in f:
config_file_list.append (line)
# Устанавливаем соединение с ECS, присваиваем переменным заначения списка
host = config_file_list[0].strip()
port = int(config_file_list[1].strip())
secure = port == int(config_file_list[1].strip())
access_key_id = config_file_list[2].strip()
secret_key = config_file_list[3].strip()
debug_level = int(config_file_list[4].strip())
calling_format = OrdinaryCallingFormat()
s3 = S3Connection(aws_access_key_id=access_key_id, aws_secret_access_key=secret_key, is_secure=secure, port=port,
host=host, debug=debug_level,
calling_format=calling_format)
Доступ к данным с помощью различных SDK
![Page 14: Откуда движется волна? EMC_ECS_v2.pdfи требований ... – Intel Dual Quad Core x86 Nodes • Dual 4-core IvyBridge (gen1) • Dual 6-core Haswell (gen](https://reader033.fdocuments.net/reader033/viewer/2022050212/5f5e9d1dae68d0169d3b5d86/html5/thumbnails/14.jpg)
© Dell Inc., 2017 г.14
Частное облако
• Lower cost than public cloud
• Unmatched combination of storage efficiency and
data access
• Anywhere read/write access with strong consistency
simplifies finding and using archived assets
• No single points of failure increases availability
and performance
• Universal accessibility eliminates storage silos and
inefficient archiving processes
• Comprehensive data types satisfy the broadest range
of application needs
![Page 15: Откуда движется волна? EMC_ECS_v2.pdfи требований ... – Intel Dual Quad Core x86 Nodes • Dual 4-core IvyBridge (gen1) • Dual 6-core Haswell (gen](https://reader033.fdocuments.net/reader033/viewer/2022050212/5f5e9d1dae68d0169d3b5d86/html5/thumbnails/15.jpg)
© Dell Inc., 2017 г.15
ECS “Storage Engine” уникальные возможности
![Page 16: Откуда движется волна? EMC_ECS_v2.pdfи требований ... – Intel Dual Quad Core x86 Nodes • Dual 4-core IvyBridge (gen1) • Dual 6-core Haswell (gen](https://reader033.fdocuments.net/reader033/viewer/2022050212/5f5e9d1dae68d0169d3b5d86/html5/thumbnails/16.jpg)
© Dell Inc., 2017 г.16
Одновременный много-протокольный доступ
16
• CAS: SDK v3.1.54
• Swift: byte range
update within an
object
• Retentions
• Keystone
integration: drop-in
replacement for
OpenStack Swift
• Primary file system
with native Ambari
2.2 integration
• Byte range
updates
• Retentions
• Metadata search
extension• NFS v3
• Global namespace
with global locking
XXXXX
![Page 17: Откуда движется волна? EMC_ECS_v2.pdfи требований ... – Intel Dual Quad Core x86 Nodes • Dual 4-core IvyBridge (gen1) • Dual 6-core Haswell (gen](https://reader033.fdocuments.net/reader033/viewer/2022050212/5f5e9d1dae68d0169d3b5d86/html5/thumbnails/17.jpg)
© Dell Inc., 2017 г.17
Elastic Cloud
Storage
S3
DATADOMAIN
DED
UPE
A B C D
A
B
C D
A
B
C D
CLOUDBOOST
CLOUDPOOLS
CENTRAL
Isilon
ROBO
DAS
CLOUDARRAY
S3S3
БЕСШОВНЫЙ АРХИВ ДЛЯ DD, CLOUD ARRAY И ISILON
CloudBoost
Долговременный
архив
CloudBoost и CloudPools
![Page 18: Откуда движется волна? EMC_ECS_v2.pdfи требований ... – Intel Dual Quad Core x86 Nodes • Dual 4-core IvyBridge (gen1) • Dual 6-core Haswell (gen](https://reader033.fdocuments.net/reader033/viewer/2022050212/5f5e9d1dae68d0169d3b5d86/html5/thumbnails/18.jpg)
© Dell Inc., 2017 г.18
Retention policy management
Retention enforcement
Data immutability
Advanced Rétention Management
Access Locks
Lock/unlock user or bucket
Meet storage requirements for
SEC 17a-4(f)
CFTC 1.31(b)-(c)
NF Z 42-013
Features
Key Benefits
Требования регуляторов
![Page 19: Откуда движется волна? EMC_ECS_v2.pdfи требований ... – Intel Dual Quad Core x86 Nodes • Dual 4-core IvyBridge (gen1) • Dual 6-core Haswell (gen](https://reader033.fdocuments.net/reader033/viewer/2022050212/5f5e9d1dae68d0169d3b5d86/html5/thumbnails/19.jpg)
© Dell Inc., 2017 г.19
Гео- распределённое хранениеВысокая надёжность с минимальными накладными расходами
19
A
A
B
B
C
A
PUT obj1 (1 MB)
PUT obj2
GET obj1GET obj1
Replicate
the 128 MB
container
XOR reductionobj1 (1 MB)
Read the
Object from
Site-1
Cache the
128 MB chunk
Read from
the cache
B
![Page 20: Откуда движется волна? EMC_ECS_v2.pdfи требований ... – Intel Dual Quad Core x86 Nodes • Dual 4-core IvyBridge (gen1) • Dual 6-core Haswell (gen](https://reader033.fdocuments.net/reader033/viewer/2022050212/5f5e9d1dae68d0169d3b5d86/html5/thumbnails/20.jpg)
© Dell Inc., 2017 г.20
Эффективное хранение
1 1.33 1.2
2 2.67 2.4
3 2.00 1.8
4 1.77 1.6
5 1.67 1.5
6 1.60 1.44
7 1.55 1.40
8 1.52 1.37
![Page 21: Откуда движется волна? EMC_ECS_v2.pdfи требований ... – Intel Dual Quad Core x86 Nodes • Dual 4-core IvyBridge (gen1) • Dual 6-core Haswell (gen](https://reader033.fdocuments.net/reader033/viewer/2022050212/5f5e9d1dae68d0169d3b5d86/html5/thumbnails/21.jpg)
© Dell Inc., 2017 г.21
2MB Buffered Writer
Эффективность хранения и производительность
21
AA
A
Store
objects in
chunks on
disksAdd parity
Delete
original
chunks
In memory
Requests
![Page 22: Откуда движется волна? EMC_ECS_v2.pdfи требований ... – Intel Dual Quad Core x86 Nodes • Dual 4-core IvyBridge (gen1) • Dual 6-core Haswell (gen](https://reader033.fdocuments.net/reader033/viewer/2022050212/5f5e9d1dae68d0169d3b5d86/html5/thumbnails/22.jpg)
© Dell Inc., 2017 г.22
Поиск по метаданным
Упрощение работы приложений
22
Save objects with metadata
Search objects using GPS coordinates, image resolution, …
/?query=x-amz-meta-image-gps-latitude>50&…
![Page 23: Откуда движется волна? EMC_ECS_v2.pdfи требований ... – Intel Dual Quad Core x86 Nodes • Dual 4-core IvyBridge (gen1) • Dual 6-core Haswell (gen](https://reader033.fdocuments.net/reader033/viewer/2022050212/5f5e9d1dae68d0169d3b5d86/html5/thumbnails/23.jpg)
© Dell Inc., 2017 г.23
Native NFS v3 capability
Rich ACLs
Global namespace
Global locking
Multi-protocol access object, NFS and HDFS
Ingest data in native format
Requires no change on the application level, accelerating the move to an object platform
Features
Key Benefits
Встроенная поддержка NFS
![Page 24: Откуда движется волна? EMC_ECS_v2.pdfи требований ... – Intel Dual Quad Core x86 Nodes • Dual 4-core IvyBridge (gen1) • Dual 6-core Haswell (gen](https://reader033.fdocuments.net/reader033/viewer/2022050212/5f5e9d1dae68d0169d3b5d86/html5/thumbnails/24.jpg)
© Dell Inc., 2017 г.24
S3 API
Caching
Multipart upload and download
Retention & versioning
ACL translation
Client side load balancing
Ingest data in native format
Requires no change on the application level, accelerating the move to an object platform
Features
Key Benefits
Поддержка CIFS
![Page 25: Откуда движется волна? EMC_ECS_v2.pdfи требований ... – Intel Dual Quad Core x86 Nodes • Dual 4-core IvyBridge (gen1) • Dual 6-core Haswell (gen](https://reader033.fdocuments.net/reader033/viewer/2022050212/5f5e9d1dae68d0169d3b5d86/html5/thumbnails/25.jpg)
SMART STORAGE- Sync and Share
- IoT and Analytics- Modern App
![Page 26: Откуда движется волна? EMC_ECS_v2.pdfи требований ... – Intel Dual Quad Core x86 Nodes • Dual 4-core IvyBridge (gen1) • Dual 6-core Haswell (gen](https://reader033.fdocuments.net/reader033/viewer/2022050212/5f5e9d1dae68d0169d3b5d86/html5/thumbnails/26.jpg)
Простое внедрение и настройка
Многопротокольный доступ (Object, File)
Интеграция с продуктами DellEMC (DD, Isilon)
Масштабируемость (зетабайты )
Гео- распределённый активный архив
Основа частного облака (Metering)
![Page 27: Откуда движется волна? EMC_ECS_v2.pdfи требований ... – Intel Dual Quad Core x86 Nodes • Dual 4-core IvyBridge (gen1) • Dual 6-core Haswell (gen](https://reader033.fdocuments.net/reader033/viewer/2022050212/5f5e9d1dae68d0169d3b5d86/html5/thumbnails/27.jpg)
https://portal.ecstestdrive.com/ -
Публичное облако (идеальный тест)
https://github.com/EMCECS -
Примеры, API и т.д.
https://www.emc.com/collateral/white-papers/h14071-ecs-architectural-
guide-wp.pdf - Архитектура