HPC for CAE with ANSYS - CADFEM UK and Ireland€¦ · HPC for CAE with ANSYS ... Recent...
Transcript of HPC for CAE with ANSYS - CADFEM UK and Ireland€¦ · HPC for CAE with ANSYS ... Recent...
HPC for CAE
HPC for CAE with ANSYS
Peter Tiefenthaler & Christof Gebhardt
CADFEM GmbH
Herbert Güttler
MicroConsult GmbH
HPC for CAE
- 1 -
High Performance Computing (HPC) at ANSYS:
The need for speed
Usage scenarios
From workstations to cloud solutions
The software side
Applications
Licensing
HPC for CAE
- 2 -
HPC Defined
High Performance Computing (HPC) at ANSYS:
An ongoing effort designed to remove
computing limitations from engineers who
use computer aided engineering (CAE) in all phases
of design, analysis, and testing.
It is a hardware & software initiative!
HPC for CAE
- 3 -
Need for Speed
Larger FE Models
Assemblies instead of single parts and defeaturing
Lots of computations!!
Mesh fidelity continues to increase
More equations to solve
More computations!!
More complex physics being analyzed
Nonlinear, Dynamics, Multiphysics
More computations!!
Parametric studies, Optimization
More computations!!
HPC for CAE
- 4 -
HPC Revolution
Recent advancements have revolutionized the
computational speed available on workstations
- Multicore processors
- Large amounts of RAM
- SSDs replacing conventional HDs
- (Multi-) GPU support
HPC for CAE
- 5 -
Workstation
Local High End Computing power
Local High End 3D Graphics
Up to 16 (24) Cores and 512 GB Memory
Benefit
All kind of sequential simulation processing
Desktop Workstation
Z1 All-in-One Workstation
Mobile Workstation
HPC for CAE
- 6 -
Compute Server
Remote High End Computing power
No 3D Graphics
Scalable in cores, memory, disks
Redundant components
Benefit
All kind of sequential and simultaneous
simulation processing
Highly scalable in the number of cores
per job
Highly scalable in the number of
simultaneous jobs large DoE‘s
HPC for CAE
- 7 -
Usage Scenario Workstation + Compute Server
Workstation + Compute-Server
Small Jobs locally on Workstation
Pre/Postprocessing locally on Workstation
Big computing jobs centrally on compute server
Load sharing useful
Simulation data
HPC for CAE
- 8 -
Usage Scenario ‚Virtual Cluster‘
HPC for CAE
- 9 -
The bandwidth bottleneck:
Workstation + Compute-Server
The amount of data sent to the
‚number cruncher‘ is usually small
Performance benefits from using
a remote server are often canibalized by
times spent for data transfer back to the
workstation prior to postprocessing.
Can be TBs of data across nodes
for cluster setup
Results 1…100GB Model 0,1..1 GB
GB..TBs
HPC for CAE
- 10 -
Blade Workstation
Workstation inside a rack (cluster system)
3D Graphics inside the rack
Redundant components
16 Graphics Workstations in C7000 enclosure
HPC for CAE
- 11 -
Usage Scenario Blade Workstations
Blade Workstations + Compute Server +
Terminals
Small Jobs + Pre/Post on Blade
Workstation
Big computing jobs on compute server
Blade Workstation graphics access by thin
clients (terminals)
Exclusive usage per user
Compression of 3D visualization data in
software (HP RGS, Citrix HDX 3D) or
hardware
Benefit
High bandwidth connection from blade
workstation to compute server fast
postprocessing
Simulation data
3D visualization data
HPC for CAE
- 12 -
Virtual Workstations
Multiple virtualized workstation on a
single physical set of hardware
Software program behaves like
hardware
Existing solutions for 3D-applications:
VMWARE, NICE DCV, Citrix
HPC for CAE
- 13 -
Usage Scenario Virtual Workstations + Compute
Server Virtual Workstations + Compute
Server + Terminals
Small Jobs + Pre/Post on Virtual
Workstation
Big computing jobs on compute server
Blade Workstation graphics access by
thin clients (terminals)
Non-Exclusive usage GPU server
Compression of visualization data in
software
Benefit
High bandwidth connection from blade
workstation to compute server fast
postprocessing
Flexible allocation of virtual
workstations cost effective
„workstation“ usage by multiple users
Simulation data
3D visualization data
0010101001110
010101001011
010100100100
1000111010110
1011110101011
101010100101
HPC for CAE
- 14 -
Terminals & Cloud
Benefit
High bandwidth connection from blade
workstation to compute server fast
postprocessing
Flexible allocation of virtual
workstations cost effective
„workstation“ usage by multiple users
Flexible scaling of hardware resources
better scaling and availability by
external hardware sharing
Simulation data
3D visualization data
HPC for CAE
- 15 -
Parallel Processing – Hardware
2 Types of memory systems
- Shared memory parallel (SMP) <= single box, workstation/server
- Distributed memory parallel (DMP) <= multiple boxes, cluster
Workstation Cluster
HPC for CAE
- 16 -
Parallel Processing - Software
2 Types of Parallel Processing
ANSYS (SMP)
- Shared memory parallel (-np > 1)
• Can only be used on single machine
Distributed ANSYS or DANSYS (DMP)
- Distributed memory parallel (-dis –np > 1)
• Can be used on single machine or cluster
• Requires 3rd party software (MPI)
Microsoft MPI on Windows HPC clusters
Platform MPI (HP MPI) on Windows workstations/servers
Platform MPI (HP MPI) on Linux workstations/servers
Intel MPI on Windows/Linux workstations/servers
HPC for CAE
- 17 -
Parallel Processing - Hardware
GPU acceleration (-acc)
- First available in v13.0 SMP using NVIDIA GPUs
- At v14.0 available for DMP
- V14.5 supports multiple GPUs
- Can be used on single machine or cluster
HPC for CAE
- 18 -
Distributed ANSYS Design Requirements
No limitation in simulation capability
Must support all features
Continually working to add more functionality with each release
Reproducible and consistent results
Same answers achieved using 1 core or 100 cores
Same quality checks and testing are done as with SMP version
Uses the same code base as SMP version of ANSYS
Support all major platforms
Most widely used processors, operating systems, and interconnects
Supports same platforms that SMP version supports
Uses latest versions of MPI software which support the latest interconnects
HPC for CAE
- 19 -
Distributed ANSYS – Domain Decomposition
• Distributed steps (‐dis ‐np N)
– At start of first load step, decompose
FEA model into N domains.
– Each domain represents a single
process and is solved on a different
core
– Solving generates a lot of data traffic!!
– Lots of communication required to
achieve solution
– All domains should be equal in size
– Each process writes its own sets of
files (file0*, file1*, file2*,…, file[N‐1]*)
– Results are automatically combined at
end of solution
HPC for CAE
- 20 -
Distributed ANSYS – Domain Decomposition
domain 0
interprocess
communication processor 1
processor 0 (host)
processor n-1
domain
decomposition
…
elem
assemble
solve
domain 1 domain n-1
elem
assemble
solve
elem
assemble
solve
elem output elem output elem output
…
combining results
HPC for CAE
- 21 -
HPC for CAE
- 22 -
HPC for CAE
- 23 -
HPC for CAE
- 24 -
HPC for CAE
- 25 -
HPC for CAE
- 26 -
HPC for CAE
- 27 -
HPC for CAE
- 28 -
HPC for CAE
- 29 -
HPC for CAE
- 30 -
HPC for CAE
- 31 -
HPC for CAE
- 32 -
HPC for CAE
- 33 -
HPC for CAE
- 34 -
HPC for CAE
- 35 -
ANSYS Remote Solve Manager
HPC for CAE
- 36 -
ANSYS Remote Solve Manager
The RSM Client machine submits jobs to a Solve Manager running
on a remote machine.
The remote machine also acts as Compute Server.
HPC for CAE
- 37 -
ANSYS Remote Solve Manager
The RSM Client machine submits jobs to a Solve Manager running
on a remote machine.
The Solve Manager then assigns the job to remote Compute Servers
(Compute Cluster)
HPC for CAE
- 38 -
ANSYS Remote Solve Manager
HPC for CAE
- 39 -
ANSYS Remote Solve Manager
HPC for CAE
- 40 -
ANSYS Remote Solve Manager
HPC for CAE
- 41 -
ANSYS Remote Solve Manager
HPC for CAE
- 42 -
ANSYS Remote Solve Manager
HPC for CAE
- 43 -
Load Sharing
Using more than 1 node for a single job with the RSM requires
using a load sharing system
Currently ANSYS 14.5 supports the following Load Sharing Systems:
Microsoft Windows HPC Server 2008 R2
LSF (Load Sharing Facility) from Platform Computing
PBS Pro from Altair
PBS also available as freeware with strongly reduced functionality
HPC for CAE
- 44 -
Load Sharing
Load Sharing means to manage many different tasks (analysis jobs),
which are allocated to workstations, compute servers and compute
clusters.
Target: Every existing jobs
shall be distributed to the
available hardware resources
via intelligent management to
get optimal performance
HPC for CAE
- 45 -
Different Licensing Models
HPC Single Licenses
HPC Packs
HPC Workgroup Solution
HPC Enterprise Solution
HPC Parametric Pack Licensing
HPC for CAE
- 46 -
HPC Single Licenses
– Flexibility for customers using less than 8 parallel per solver seat and less than 128 parallel overall.
– 2 Cores for free in ANSYS basic license are added to the single HPC licenses available.
For example if customer buys 2 single HPC licenses, the ANSYS run can be started with 4 Cores parallel
HPC for CAE
- 47 -
HPC Packs
Release
4.5
HPC for CAE
- 48 -
HPC Workgroup Solution
HPC for CAE
- 49 -
HPC Enterprise Solution
HPC for CAE
- 50 -
HPC Parametric Pack Licensing
HPC for CAE
- 51 -
Conclusion
HPC offers many benefits for CAE
Applications ranging from workstations to cloud solutions
The framework (workbench / RSM) is established for many years
ANSYS supports a wide range of usage scenarios