The Pennsylvania State University

The Graduate School

College of Engineering

INTERACTIVE VIRTUAL ENVIROMENT FOR DESIGN AND

MANIPULATION OF TOOLPATHS FOR USE WITH 3 DEGREE OF FREEDOM AND

5DOF BIOPRINTING PLATFORMS

A Thesis in

Engineering Science and Mechanics

by

Adomas Povilianskas

©2019 Adomas Povilianskas

Submitted in Partial Fulfillment

of the Requirements

for the Degree of

Master of Science

May 2019

ii

The thesis of Adomas Povilianskas was reviewed and approved* by the following:

Ibrahim T. Ozbolat

Associate Professor of Engineering Science and Mechanics

Thesis Advisor

Elzbieta Sikora

Associate Teaching Professor of Engineering Science and Mechanics, Graduate Programs

Coordinator

Corina Drapaca

Associate Professor of Engineering Science and Mechanics, Graduate Programs Officer

Judith A. Todd

Professor of Engineering Science and Mechanics

Department Head of Engineering Science and Mechanics

Chair of Committee

*Signatures are on file in the Graduate School

iii

ABSTRACT

Designer software interface simplifies toolpath manipulation tasks by abstraction of

complicated calculations through an interactive 3D virtual environment. Implementation of

complete bioprinting toolchain enables researchers without technical background such as doctors

and biologists to produce bioconstructions rapidly using 3 Degrees of Freedom (DOF) and 5

DOF bioprinting platforms.

iv

TABLE OF CONTENTS

List of Figures .............................................................................................................. v

List of Tables ............................................................................................................... vi

Chapter 1 Introduction ................................................................................................. 1

Development of 5-axis motion stage development .............................................. 3

Development of motion control software framework .......................................... 4 Motion control software description ..................................................................... 6

Chapter 2 Current Toolpath Generation Software Used in Bioprinting ...................... 7

Conventional 3D printing toolpath generation ..................................................... 7

2D bioprinting cell culture specific toolpath generation ...................................... 8 3D bioprinting toolpath generation ....................................................................... 9

Chapter 3 Development Requirements of Interactive 3D User Interfaces .................. 10

A-Code implementation and lessons learned ....................................................... 10 Design requirements of a virtual environment for toolpath planning ................... 11

Chapter 4 Toolpath Editor Development ..................................................................... 14

Toolpath editor framework development ............................................................. 15

Toolpath waypoints and paths .............................................................................. 16 Toolpath manipulation .......................................................................................... 16

Visual toolpath representation .............................................................................. 20 Toolpath generation of common geometry ........................................................... 23 3D scanning .......................................................................................................... 26

Chapter 5 Advanced Features ...................................................................................... 28

Toolpath generation within convex surface geometry .......................................... 28 Toolpath generation on free-form surfaces ........................................................... 31

Chapter 6 Toolpath Designer Application Examples .................................................. 37

Designs based on imported geometrical data ....................................................... 37

Toolpath generation based on 3D scan data of live specimen .............................. 38

Chapter 7 Conclusions ................................................................................................. 39

BIBLIOGRAPHY................................................................................................. 40

v

LIST OF FIGURES

Figure 1.1 Bioprinting Toolchain Implementation. ..................................................... 2

Figure 1.2 5DOF bioprinter CAD model and design. .................................................. 3

Figure 1.3 Motion control software framework implementation. ................................ 4

Figure 1.4 Motion control software user interface ....................................................... 5

Figure 2.1 PetriPrinter g-code editing interface. .......................................................... 8

Figure 3.1 A-code interface software (A) and algorithm implementation (B). ........... 10

Figure 4.1 Toolpath Editor, (A) editor UI, (B) final render of resulting structure. ..... 14

Figure 4.2 Example G-code output .............................................................................. 15

Figure 4.3 Waypoint and Path translation using graphical user interface. .................. 17

Figure 4.4 Properties and configuration of designer workspace.................................. 20

Figure 4.5 Parametrization strategies of Catmull-Rom curves. ................................... 21

Figure 4.6 Barry and Goldman’s cubic Catmull-Rom curve formulation. .................. 22

Figure 4.7 Waypoint Generators .................................................................................. 23

Figure 4.8 Example procedure for 3D scanning of a 5mm defect during surgery....... 27

Figure 5.1 Toolpath generated based on 3D scan obtained data.................................. 28

Figure 5.2 2nd degree 3x6 grid NURBS surface formulation23. ................................... 31

Figure 5.3 Knot vector generation algorithm. ............................................................. 33

Figure 5.4 Example of 10x10 (A) and 3x10 (B) NURB surfaces of degree 2. ............ 33

Figure 5.5 NURBS surface positioned tangent to the surface of an imported mesh. .. 34

Figure 5.6 NURBS surfaces of degree 2, 3 and 4. ....................................................... 35

Figure 5.7 Path generation using NURB surfaces. ...................................................... 36

Figure 6.1 Design of semilunar heart valve based on imported STL geometry. ......... 37

Figure 6.2 Testing convex toolpath generator for bioprinting on live specimen. ........ 38

vi

LIST OF TABLES

Table 3.1 Variables required for one waypoint using 3DOF and 5DOF bioprinters. .. 12

Table 3.2 Virtual environment interaction requirements. ............................................ 12

Table 3.3 Virtual environment implementation aspects. ............................................. 13

Table 4.1 Waypoint and path manipulation command list. ......................................... 18

Table 4.2 Available toolpath generators. ..................................................................... 24

1

Chapter 1 Introduction

Bioprinting is a rapidly growing multidisciplinary field of study. Bioprinting enables

researchers to produce biostructures cheaper and faster while allowing for greater flexibility in

bioconstruction design when compared to traditional manufacturing methods such as casting1.

This technology allows patterning of living cells, biological macromolecules, and other

biomaterial without significant reduction in cell viability2. Recent advances in additive

manufacturing are being adopted for bioprinting applications. According to the study conducted

by Grand View Research, this industry is projected to reach $2.6 billion market share by 20243.

Extrusion-based bioprinting originated from the fused deposition modeling (FDM)

fabrication technique created in the late 1990s. FDM was adopted to produce porous scaffolds

which provide the cells with a structure to grow and attach within, prolonging their lifespan4.

Tissue models and bioconstructs are bioprinted by slicing a solid mesh which can be

designed using Computer Aided Drafting (CAD) software or obtained by imaging techniques

such as MRI, CT or 3D scanning 5 6. Obtained slices of the geometry are used to generate

toolpaths, which the bioprinter executes layer by layer to build the desired bioconstruct.

Since bioprinting technology is still relatively new, researchers have adopted advances in

hardware and software development directly from the 3D printing industry. While large scale

models can be easily handled by conventional 3D printing slicing software7, they lack precision

control of the toolpath which is essential when printing small submillimeter scale features of

biodevices. In order to produce complicated designs, while maintaining precision control of the

2

bioprinters end effector, researchers resort to writing the G-code themselves2. This requires

calculation and definition of every waypoint for the bioprinter to execute. While this approach

works for small devices composed of few paths and waypoints, this technique requires

specialized knowlage of G-code specification, tends to be time consuming, and is prone to

mistakes. In order to simplify the biostructure design process, making bioprinting technology

accessible to researchers without technical backgrounds, new software tools are required, as

shown in the Bioprinting Toolchain Implementation schematic shown in Figure 1.1.

Figure 1.1 Bioprinting Toolchain Implementation.

3

Development of 5-axis motion stage development

Bioprinting complex scaffolds and biostructs using conventional 3-axis bioprinting

platforms is a challenging and sometimes even an impossible task8. In order to overcome these

constraints, a 5-axis 5 degrees of freedom (DOF) motion stage was designed and fabricated as

shown in Figure 1.2.

Figure 1.2 5DOF bioprinter CAD model and design.

The 5 DOF bioprinting platform features an open build surface allowing for bioprinting

directly on large specimens, by placing the device above the deposition target, such as an animal

or a human head. This thesis is an extension of ongoing efforts to implement a complete

bioprinting hardware platform and software toolchain designed specifically for bioprinting

applications. Continued work focuses on improving the software implementation of toolpath

planning and generation algorithms.

4

Development of motion control software framework

Motion control software was implemented using Python programing language enabling

the use of advanced features including loops, conditional statements, real-time monitoring of the

motion stage and peripheral device such as sensors, servo motors and solenoid valves. The

structure of the motion control software framework is summarized Figure 1.3.

Figure 1.3 Motion control software framework implementation.

Hardware subsystem handles communication with motion stages and sensors.

Automation subsystem generates G-code commands and monitors the hardware subsystem state.

Data objects contain models of reusable data. Utilities subsystem provides common functions. UI

subsystem provides graphical user interface to control and monitor the framework.

5

This framework was implemented in order to expand bioconstruction design freedom and

facilitate automation of common tasks. The G-code standard is interpreted sequentially by the

motion control card where G-code commands are translated to electrical pulses which move the

motion stage motors. The G-code standard is limited and lack logic commands such as loops and

conditional statements. A-code programing language was designed and implemented, extending

the capabilities of the G-code standard. A-code acts as an intermediate execution layer which

generates G-code commands in real-time implementing execution logic and allowing creation of

algorithm used for automation of common tasks. However, lack of an interactive user interface

and high level of complexity limited the functionality to users with technical backgrounds who

are familiar with the G-code and A-code standards. In order to reduce these technical barriers

and usage complexity; a toolpath designer virtual environment was implemented extending the

capabilities of motion control software shown in Figure 1.4.

Figure 1.4 Motion control software user interface

6

Motion control software description

Motion control software implements 3DOF, 5DOF and secondary motion stage

navigation interfaces displayed in Figure 1.4(1,2,3). These functions provide the ability to

control the position of the end effector within the build volume of the bioprinter. End effector

position may be recorded, providing quick navigation to regularly used locations as displayed in

Figure 1.4(4). Sensor and camera interfaces pictured in Figure 1.4(5,6) allow for real-time

monitoring and control of peripheral sensor devices. Automation interface shown in Figure

1.4(7) monitors the A-code subroutine execution and allows for development of A-code

algorithms in order to automate commonly performed tasks. Diagnostics interface display

information, warning and error messages. In addition, it provides a command terminal of the

primary motion stage hardware controller as shown in Figure 1.4(8).

7

Chapter 2 Current Toolpath Generation Software Used in Bioprinting

Conventional 3D printing toolpath generation

Slicing software designed for 3D printers can be used for bioprinting applications by

applying postprocessing algorithms to enable operation of pneumatic syringe dispensers7,

pressure sensors, pneumatic solenoid valves, and other peripheral devices. These software

packages tend to be complicated allowing for over 200 customizable settings9 10 11 which control

the shape of the toolpath produced by selecting layer height, path spacing, outer shell layer

count, infill direction and density to name a few. Some of the most popular slicers packages used

by the bioprinting community include Skeinforge 11, KISSlicer 12 and Slic3r 9.

While these slicers are excellent for producing toolpaths of relatively large models using

thermoplastics, they lack precision control over individual paths and waypoints. Bioprinting of

low viscosity bioinks like polydimethylsiloxane (PDMS)7 may require curing in order to obtain

optimal mechanical properties. This may result in structure buckling and printing failure during

unoptimized travel moves and path execution sequences.

8

2D bioprinting cell culture specific toolpath generation

Recently developed PetriPrinter software package aims to address some of these issues13.

PetriPrinter features Python application programming interface (API) layer in order to extend the

G-code functionality by providing toolpath generators of common shapes, live monitoring of

bioprinter states, ability to print on petri dish arrays, and to edit the g-code text imported from

third-party slicers. This software package is aimed at 2D cell culture work and does not provide

an interactive interface to manipulate toolpaths in 3D enviroment. Toolpath editing interface is

displayed in Figure 2.1.

Figure 2.1 PetriPrinter g-code editing interface.

9

3D bioprinting toolpath generation

Bioprinting specific toolpath generation tools are available, however they are designed

for specific bioprinter hardware such as Tissue Structure Information Modeling TSIM 14

developed by Advanced Solutions LLC. TSIM provides a graphical user interface for toolpath

creation, manipulation and post-processing of medical image data. However, this software works

only with a BioAssemblyBot bioprinter which costs over $100,000. High price and hardware

restrictions limit the software accessibility to a small portion of the bioprinting community.

10

Chapter 3

Development Requirements of Interactive 3D User Interfaces

A-Code implementation and lessons learned

Motion control software has been under development since 2017 and was extended to

accommodate 5-axis bioprinter platform’s motion control system8 in 2018. Motion control

software features an intermediate execution layer which allows for manipulation of toolpaths and

automation of common tasks. Automation is accomplished by implementing an intermediate

bioprinter control language A-code which interprets the commands and generates the G-code in

real-time for execution by the bioprinter. User interface displayed in Figure 3.1(A) was

implemented using (windows, icons, menus, pointing) WIMP UI model.

Figure 3.1 A-code interface software (A) and algorithm implementation (B).

11

A-code enables the use of advanced functions such as loops, conditional statements, real-

time monitoring of the motion stage and computer vision systems. In addition, it provides the

ability to interface peripheral devices such as solenoid valves and pressure sensors allowing the

user to implement automation algorithms for performing common tasks quickly. Example of A-

code algorithm flow chart is displayed in Figure 3.1(B)

While this functionality provided the means of automating everyday tasks, the resulting

system became too complicated to be useful. This was due to the lack of an interactive user

interface. Even technically inclined users would prefer to perform the tasks manually, rather than

defining an algorithm to automate repeatable tasks.

In order to overcome this issue, a user-friendly, interactive virtual environment VE was

required. Abstraction of the complicated software features was essential for broadening the user

base, making advanced features accessible for people without a technical background in

engineering such as surgeons and biologists, who dominate the bioengineering field.

Design requirements of a virtual environment for toolpath planning

The WIMP (windows, icons, menus, pointing) user interface (UI) model was developed

in the early 1980s. This UI model is excellent for applications requiring text entry and

manipulation of variables but cannot relay information such as a visual representation of

waypoints and geometry in 3D space. Toolpath planning for bioprinting applications requires the

user to define five to eight variables for every waypoint contained within the path as shown in

Table 3.1. Complicated bioconstructs may contain hundreds of paths resulting in an

overwhelming number of waypoints and variables which must be defined by the user.

12

Table 3.1 Variables required for one waypoint using 3DOF and 5DOF bioprinters.

3 DOF Bioprinting 5 DOF Bioprinting

Position vector (x,y,z) End effector velocity Volumetric flow of material

Position vector (x,y,z) End effector normal orientation vector (x,y,z) End effector velocity Volumetric flow of material

By combining the WIMP UI model with an interactive 3D virtual environment, this

complexity can be reduced significantly. A virtual environment enables the user to manipulate

and validate changes in the toolpath visually. By dragging and clicking the waypoints and paths

within the 3D space, the user’s productivity can be increased significantly. Since waypoints are

related though the parent path, all the variables defining the waypoints within the path can be

calculated automatically. The resulting paths can be inspected visually reflecting the changes

based on the users input by translating a waypoint or path within the virtual environment. Great

care must be considered when designing applications which use 3D virtual environments. People

have difficulty understanding virtual 3D spaces presented on 2D screens, due to the lack of

visual cues and depth present in the real world15. The interaction requirements are outlined in

Table 3.2.

Table 3.2 Virtual environment interaction requirements.

Requirements Description

Orbital camera control User is free to rotate, pan and zoom the perspective camera freely to inspect the design within the virtual build volume of the bioprinter.

Translation controls User is free to precisely translate waypoints, paths, and imported 3D geometry along individual axis: X, Y, Z as well as planes XY, YZ, XZ within the build volume with respect to world coordinates.

13

Table 3.2 Virtual environment interaction requirements (continued)

Requirements Description

Rotation controls User is free to orient the imported 3D geometry

Toolpath generator configurators User is free to place commonly used structures within the virtual environment with minimal effort and instant visual feedback to validate the generator’s parameters.

Based on these requirement, different aspects of the implementation should be considered

and include: Input/Output (I/O), navigation, toolpath manipulation, and selection. These features

should follow the general guidelines of interactive design maintaining simplicity, consistency

and error prevention16. These user interface aspects are outlined in Table 3.3.

Table 3.3 Virtual environment implementation aspects.

Aspect Description

IO Software should be platform independent in order to broaden the target audience and work on both desktop and mobile devices. It is expected that the device will have the following discrete inputs: mouse; and keyboard. Featuring a semi-immersive resolution independent display as an output.

Navigation and toolpath manipulation User should have the freedom of exploring the environment16 while transitioning smoothly between different interaction modes such as manipulation of waypoints, paths and imported geometry.

Selection User should be able to select specific objects within the environment as well as direction/plane of action while manipulating the objects within the virtual environment.

14

Chapter 4 Toolpath Editor Development

Figure 4.1 Toolpath Editor, (A) editor UI, (B) final render of resulting structure.

The toolpath editor allows for precise manipulation of waypoints and paths to achieve

precise and repeatable control of material deposition using a bioprinting platform. Sets of

waypoints are grouped in separate paths which are converted to A-code or G-code commands to

be executed sequentially by the bioprinter to produce the desired structure (Figure 4.1).

15

Toolpath editor framework development

Toolpath editor is written in JavaScript. JavaScript was chosen for toolpath editor

interface development due to platform independence, ease of distribution and vast availability of

common / scientific libraries17. Rendering of 3D geometry is facilitated through Three.js18

library. Mathematics libraries Verb.js19 and Math.js20 are utilized for performing mathematical

calculations. In addition, the toolpath editor may be hosted on a stand alone web server, or

served directly by the bioprinter’s interface software, where a real-time communication channel

between the bioprinter control software and toolpath editor is established through WebSocket

protocol. WebSocket allows for instantaneous monitoring of the bioprinter’s current position in

relation to the rest of the toolpaths to be executed. Currently, the designer interface is closely

coupled with the custom hardware implementation of 5 DOF motion stage, and 5 DOF toolpaths

are produced using A-code commands. 3 DOF toolpaths may be generated using G-code

standard and may be executed by any G-code compliant CNC motion stage as shown in Figure

4.2.

Figure 4.2 Example G-code output

16

Toolpath waypoints and paths

Waypoints are parametricized with a position vector p and an orientation vector n, to

orient the bioprinter’s end effector within the build volume. Waypoints also contain additional

information describing the bioink and end effector’s motion. Additional parameters include:

deposition method, end effector speed, material properties, and sequential execution information.

Sets of waypoints are represented as paths, the bioprinter will deposit the material

between the waypoints contained within the path. These paths are visualized in the editor

interface by extruding a circular shape along the path defined by a Catmull-Rom spline

constrained through the waypoints within the path.

The resulting design may contain many paths which are separated by travel moves during

which material deposition is turned off. Waypoints, paths and travel moves are displayed in

Figure 4.1.

Toolpath manipulation

Waypoints and paths can be translated, reversed and deleted within the 3D environment

to design and modify the desired structure (Figure 4.3). In order to simplify the design process,

newly appended waypoints snap to the previously placed waypoints and build platform surface.

The build platform surface is divided by an equally spaced grid pattern spanning the diameter of

the filament strand. The Filament diameter can be changed at any point during the design process

in order to accommodate different bioinks and biostruct design requirements. This approach

makes it easy to prototype symmetrical and dimensionally accurate devices quickly. Rendering

17

of the resultant toolpath is instantaneous; providing the user with visual feedback reflecting the

changes within the toolpath geometry.

Figure 4.3 Waypoint and Path translation using graphical user interface.

Toolpath tree manipulation and sorting techniques are summarized below in Table 4.1.

While the order of executed commands may be arbitrary, the final structure depends on the

command execution order.

18

Table 4.1 Waypoint and path manipulation command list.

Blue – start, Gray – middle, Red – ending

Structure reversal

Reverses all paths within the device, where the starting waypoint becomes the final waypoint.

Path path reversal Reverses a specific path, where the starting waypoint becomes the final waypoint.

Path deletion

Deletes a specific path within the bioconstruct.

Waypoint material Sets path parametrization and visual appearance for specific bioink type.

Path translation Translates all the waypoints contained within the path to the new location.

19

Table 4.1 Waypoint and path manipulation command list. (continued)

Blue – start, Gray – middle, Red – ending

Waypoint deletion

Deletes specific waypoint within a path.

Waypoint translation Translates the waypoint to the new location.

Waypoint speed Sets new end effector velocity for all further paths and waypoints, including the selected waypoint.

Travel move configuratrion

Prescribes the travel move strategy between the paths. Hop mode – bioprinter lifts the nozzle between the ending waypoint of the previous path and starting waypoint of sequential path. Flat mode - bioprinter does not lift the nozzle between the ending waypoint of the previous path and starting waypoint of sequential path.

20

Visual toolpath representation

The graphical user interface enables configuration of the designer workspace to

accommodate the biostruct design requirements. In order to reduce the complexity of workspace

and tool configuration, most of the variables and functions are derived from the following

independent variables: nozzle diameter; hop height for travel moves; build platform size;

material type; and starting position. These settings may be changed at any point throughout the

design process. Different configurations are displayed in Figure 4.4.

Figure 4.4 Properties and configuration of designer workspace

Configuring the hop height and starting position (A). Configuring the nozzle diameter (B).

21

Approximation of resulting structure is performed by constructing a circular extrusion

along the path by fitting a cubic centripetal Catmull-Rom spline through the waypoints contained

within the path. The tightness of the spline curve may be controlled through knot parametrization

variable α defined between 0 and 1. In addition, the overall shape of the path is also dependent

on waypoint density as displayed in Figure 4.5. Mathematical formulation of a Catmull-Rom

spline is presented in Figure 4.6.

Figure 4.5 Parametrization strategies of Catmull-Rom curves.

22

Figure 4.6 Barry and Goldman’s cubic Catmull-Rom curve formulation.

Curve segment C is defined between control points 𝑃0−3 and knot values 𝑠0−3 which are

parametrized through variable α. Intermediate coefficients L are used to produce interpolating 3rd Degree

polynomial curve segments which are valid between neighboring control points P.21

Low values of α produce a tighter curve where α=0, α=1 corresponds to uniform (red)

and chordal (blue) parametrization strategies as shown in Figure 4.5. Centripetal parametrization

of a Catmull-Rom spline was chosen, because it produces a smooth curve between the waypoints

without cusps and self-intersections21. This parametrization strategy provides greater control

over the shape of the path in order to match the shape of different bioinks with varied material

properties.

Values of α between 0.05 and 0.2 were determined to be most appropriate for

representing the extrusion of low viscosity materials such as PDMS and Pluronic by visual

inspection. Color and opacity of the filament may be adjusted to match the visual characteristics

of the bioink.

23

Toolpath generation of common geometry

Toolpath generation for common parametric shapes such as planes, cylinders, cones,

spirals and boxes are provided through an interactive user interface (Figure 4.7).

Figure 4.7 Waypoint Generators

The desired shape’s dimensions are defined by selecting the starting and ending points of

the shape. Additional shape dependent parameters such as height, wall thickness and toolpath

direction must be defined though WIMP UI interface. Toolpath generators allow the user to

prototype quickly, automating the waypoint placement process. The filament diameter is used to

calculate optimal waypoint placement, which is compatible with other tools and manually placed

waypoints allowing them to be used in combination to produce complex designs. Before the

waypoints are generated, a convex hull of the shape is displayed to validate the selected

parameters. Currently available generators are listed below in

Table 4.2. Generators are implemented to produce toolpaths which are continuous and do

not require any travel moves to produce the generated shape. This is important, because travel

moves have a potential of introducing defects and can cause buckling of the structure as

discussed previously.

24

Table 4.2 Available toolpath generators.

Generator Shape parameters Use cases

Snake

Snake generator produces a snake pattern within a rectangular bounding box.

•Pattern direction •Evenly cover large areas •Top and bottom faces of rectangular structures

Box

Box generator produces a rectangular box pattern.

•Height •Wall count

•Perfusion boxes •Outer shells •Channels •Solid square beams

Spiral

Spiral generator produces a spiral pattern.

•Start point •Direction

•Top and bottom faces of circular structures •Even filling of circular areas •Channels •Contact points •Solid rods

25

Table 4.2 Available toolpath generators. (continued)

Generator Shape parameters Use cases

Cylinder/Cone

Cylinder generator produces a cylindrical pattern.

•Start point •Top taper •Bottom taper •Wall segment count •Height

•Circular containers •Channels •Contact points •Hollow rods

Convex Mesh

Convex Mesh generator produces toolpaths outlining sliced mesh geometry.

•Mesh, position, orientation •Minimum and Maximum radius of geometry •Layer count •Initial nozzle size •Loop count •Neighboring point trim •Waypoint spacing

•Even filling of simple arbitrary shapes •Repeatable complex toolpath generation

Surface Generator

Surface Generator produces toolpaths along a NURBS surface.

•Grid size, spacing •Degree of surface •Waypoint spacing

•Curved geometry •Channels •Coating of arbitrary surfaces

26

3D scanning

The ability to visualize, manipulate and produce toolpaths based on captured mesh

surfaces is a desirable feature in bioprinting community.22 In order to produce anatomically

correct devices and models, a precise virtual representation of their internal structures and

surface topologies may be obtained through medical imaging techniques including CT, MRI and

3D scanning as shown in Figure 4.8.

3D scanners produce point cloud representations of captured surfaces. Point cloud data

can be noisy and contain artifacts as displayed in Figure 4.8(B). Post-processing techniques must

be selected on an individual basis to accommodate the present scanning conditions in order to

produce a valid mesh comprised of triangular faces defined by a set of vertices as shown in

Figure 4.8(D). One of the most commonly used standards for captured geometry representation

is stereolithography (STL) format specification.

First, the specimen and targeted scanning areas must be cleaned thoroughly as presented

in Figure 4.8(A). Any liquid contained within the concavities of the scan target area will produce

surface artifacts due to diffraction of the scanner’s laser beam, as shown in Figure 4.8(D).

Immediately after cleaning, the subject is scanned using a handheld Artec Spider 3D scanner

(Artec3D, usa). The obtained point cloud is registered, aligned and converted to a solid mesh

composed of triangles and vertices. This mesh is exported in STL format to be used within the

toolpath designer environment. An imported mesh can serve as a reference while designing the

biostruct, or can be used to produce toolpaths within the concavities using convex mesh

generator.

27

Figure 4.8 Example procedure for 3D scanning of a 5mm defect during surgery.

In this example live rat specimen is prepared by the surgeon. Exposed cranium of the rat

is 3D scanned before bioprinting of hydrogel bioink containing cells which promote bone

regeneration. Four regions are shown in Figure 4.8. Region A show the exposed skull of a rat

with two critical (5mm) defects. Region B show the initial point cloud captured using 3D

scanner. Region C show the point cloud data after application of post-processing techniques.

Region D show the final geometry ready for import within designer virtual environment.

28

Chapter 5 Advanced Features

Toolpath generation within convex surface geometry

Bioprinting directly on live specimen is challenging due to variables such as surface

topology, contour and convexity of the surface, a bioprinter’s nozzle should not collide with the

specimen. In addition, there is a time constraint, because the live specimen is under anesthesia

during bioprinting operation and may become alert. While it is feasible to perform the waypoint

placement manually, the number of required waypoints grow quickly when the size of the defect

increases. In order to overcome these issues an automatic generation algorithm was developed

and implemented as a shape generator, as shown in Figure 5.1. The principles of operation and

use requirements for this shape generator are explained below.

Figure 5.1 Toolpath generated based on 3D scan obtained data.

Seven regions are shown in Figure 5.1. Region A show the 3D scan of a rat’s cranium

imported within designer virtual environment. Region B show the obtained contour of the

defect’s geometry. Region C show the visual validation of the toolpath produced using convex

mesh generator. Region D show the waypoints produced using convex mesh generator. Region E

show the final complete toolpath ready for bioprinting. Regions F and G show the toolpath

produced using convex mesh generator within complex defect boundary with different

parametrization strategies.

29

First, subject’s geometrical representation is obtained using a handheld 3D scanner. Solid

mesh is imported within a virtual environment, where it is prepared for toolpath generation by

aligning the mesh within the printers build volume as shown in Figure 5.1(A).

Next, the mesh geometry is sliced using an xy-plane defined at the origin of build

surface, as shown in Figure 5.1(B). The slicing is performed by checking for intersections

between the plane and the lines connecting vertices of triangular faces. Intersection points are

aggregated into an unordered list defining the wall boundary of the mesh along the slicing plane.

The desired contour is then chosen by clicking the boundary outline as shown in Figure

5.1(B). Since meshes may contain numerous independent islands, the user must select the desired

contour. A nearest-member search algorithm is used for a selected contour point to define an

ordered list of points along the contour. A Catmul-rom spline is fit through the ordered set of

points to obtain a smooth curve.

Finally, the generator’s parameters can be adjusted in order to modify the generated

waypoint density until the desired shape of the toolpath is obtained and visually validated as

shown in Figure 5.1(C). The previously obtained smooth curve defining the boundary of the

geometry is used to generate the first path along the geometry’s border by offsetting the curve

inwards in relation to the direction of the path. Sequential loops use the previous path’s smooth

curve to generate up to Lc contours. While this algorithm is capable of handling convex circular

curves very well as shown in Figure 5.1(E), self-intersections begin to arise along concavities

throughout the path as seen in Figure 5.1(G). This problem is addressed by adjusting waypoint

spacing and checking for neighboring points before subsequent waypoint placement. This may

be adjusted through algorithm parametrization as shown in Figure 5.1(F). A pseudo code for the

base toolpath generation algorithm is provided in Figure 5.2.

30

Pseudo code:

1 𝐶𝑝 = 𝐶𝑖

2 for each 𝐿𝑐 3 for each 𝑊 in 𝐶𝑝

4 𝑾𝒑 = 𝑾𝒑 + 𝑛𝑜𝑟𝑚(𝑪𝒅 × 𝑺𝒏 ) ∗ 𝐷𝑛

5 end 6 𝐶𝑝 = 𝐶𝑐

7 end

Variable definitions:

𝐶𝑖 = 𝐼𝑛𝑖𝑡𝑖𝑎𝑙 𝑝𝑎𝑡ℎ,

𝐶𝑝 = 𝑃𝑟𝑒𝑣𝑖𝑜𝑢𝑠 𝑝𝑎𝑡ℎ,

𝐶𝑐 = 𝐶𝑢𝑟𝑟𝑒𝑛𝑡 𝑝𝑎𝑡ℎ, 𝑪𝒅 = 𝐶𝑢𝑟𝑟𝑒𝑛𝑡 𝑝𝑎𝑡ℎ 𝑑𝑖𝑟𝑒𝑐𝑡𝑖𝑜𝑛 𝑣𝑒𝑐𝑡𝑜𝑟,

𝐿𝑐 = 𝐿𝑜𝑜𝑝 𝑐𝑜𝑢𝑛𝑡, 𝐷𝑛 = 𝐷𝑖𝑎𝑚𝑒𝑡𝑒𝑟 𝑜𝑓 𝑡ℎ𝑒 𝑛𝑜𝑧𝑧𝑙𝑒,

𝑊 = 𝑊𝑎𝑦𝑝𝑜𝑖𝑛𝑡, 𝑾𝒑 = 𝑃𝑜𝑠𝑖𝑡𝑖𝑜𝑛 𝑜𝑓 𝑐𝑢𝑟𝑟𝑒𝑛𝑡 𝑊𝑎𝑦𝑝𝑜𝑖𝑛𝑡,

𝑺𝒏 = 𝑁𝑜𝑟𝑚𝑎𝑙 𝑣𝑒𝑐𝑡𝑜𝑟 𝑜𝑓 𝑠𝑙𝑖𝑐𝑖𝑛𝑔 𝑝𝑙𝑎𝑛𝑒,

Figure 5.2 Pseudo code of toolpath generation algorithm within convex surface geometry

31

Toolpath generation on free-form surfaces

Bioprinting along complex surface geometries is challenging, since a stereolithography

(STL) geometry of a surface representation may contain an unordered list of vertices and faces,

and there may be discontinuities across the surface, missing geometry, roughness, and acute

interfacial angles, which impact the stability of toolpath planning algorithms. These issues may

result in large changes of surface normal vector producing paths outside the capabilities of the

bioprinter’s hardware. In order to overcome this problem, we need a surface representation

which is defined by smooth differentiable functions. These functions are used for finer

segmentation of paths to produce waypoints positions and normal vectors within the capabilities

of the bioprinters hardware.

Nonuniform rational B-spline (NURBS) representation of a surface 𝑆(𝑢, 𝑣) can satisfy our

requirements23. In order to implement a dynamic 𝑁𝑤 x 𝑁ℎ grid surface with desired spacing (𝑆𝑝),

we must generate an appropriate set of control points (P), weights (w), degrees (n) and knot

vectors (k) to satisfy the basis functions (N), Graphical representation of mathematical 2nd degree

NURBS surface formulation is shown in Figure 5.2.

Figure 5.2 2nd degree 3x6 grid NURBS surface formulation23.

32

𝑆(𝑢, 𝑣) = ∑ ∑ 𝑅𝑖,𝑛(𝑢, 𝑣) 𝑃𝑖,𝑗

𝑙

𝑗=1

𝑘

𝑖=1

𝑅𝑖,𝑗(𝑢, 𝑣) = 𝑁𝑖,𝑛(𝑢)𝑁𝑗,𝑚(𝑣)𝑤𝑖,𝑗

𝑁𝑝,𝑛(𝑢)𝑁𝑞,𝑚(𝑣)𝑤𝑝,𝑞

𝑁𝑖,𝑛 = 𝑓𝑖,𝑛𝑁𝑖,𝑛−1 + 𝑔𝑖+1,𝑛𝑁𝑖+1,𝑛−1

𝑓𝑖,𝑛(𝑢) =𝑢−𝑘𝑖

𝑘𝑖+𝑛−𝑘𝑖, 𝑔𝑖,𝑛(𝑢) =

𝑘𝑖+𝑛−𝑢

𝑘𝑖+𝑛−𝑘𝑖

𝑓𝑖,𝑛(𝑣) =𝑢−𝑘𝑖

𝑘𝑖+𝑛−𝑘𝑖, 𝑔𝑖,𝑛(𝑣) =

𝑘𝑖+𝑛−𝑢

𝑘𝑖+𝑛−𝑘𝑖

Function 𝑆(𝑢, 𝑣) represents the NURBS surface which is bivariate vector-valued piecewise

rational function defined in region (𝑢, 𝑣) ∈ [0, 1]. where 𝑅𝑖,𝑗(𝑢, 𝑣) define piecewise rational basis

functions of desiered degree 𝑛 defined on knot vectors 𝑘𝑢 𝑎𝑛𝑑 𝑘𝑣 which are nondecreasing sequences of

real numbers. Knot vectors control the span of peacewise functions though control point net defined by

𝑃𝑖,𝑗 where i,j corespond to 𝑁𝑤 x 𝑁ℎ grid surface with desired spacing (𝑆𝑝) 23.

The knot vector must be uniform while clamping the surface at the surrounding edge

control points. Since the shape and size of the captured geometry may vary, flexibility of the

surface size and control point density is required. Therefore, the knot vector must be computed

automatically and depend on the grid size. The Pseudo code for knot vector generation is

presented in Figure 5.3.

33

Pseudo code:

1 2 3 4 5 6

7 8

𝑓𝑜𝑟 𝑒𝑎𝑐ℎ 𝑘𝑢 𝑖𝑓 𝑖 ≤ 𝑛𝑢 𝑘𝑖 = 0 𝑒𝑙𝑠𝑒 𝑖𝑓 𝑖 ≥ 𝑁𝑤 𝑘𝑖 = 1 𝑒𝑙𝑠𝑒 𝑘𝑖 = 𝑖/ 𝑁𝑤 end

𝑓𝑜𝑟 𝑒𝑎𝑐ℎ 𝑘𝑣 𝑖𝑓 𝑖 ≤ 𝑛𝑣 𝑘𝑖 = 0 𝑒𝑙𝑠𝑒 𝑖𝑓 𝑖 ≥ 𝑁ℎ 𝑘𝑖 = 1 𝑒𝑙𝑠𝑒 𝑘𝑖 = 𝑖/𝑁ℎ end

Figure 5.3 Knot vector generation algorithm.

Selected examples of different surfaces and control point configurations are provided in

Figure 5.4. 2nd Degree surfaces feature high local surface dependence on control point position as

shown in Figure 5.4(A). Four neighboring control points are enough to position the surface

through the plane defined by the four control points. In addition, small grids spanning only three

control points may be used as shown in Figure 5.4(B).

Figure 5.4 Example of 10x10 (A) and 3x10 (B) NURB surfaces of degree 2.

Example 1

10x10 surface of degree 2 is presented in Figure 5.4(A):

𝑘𝑢 = [0, 0, 0, 0.23, 0.30, 0.38, 0.46, 0.53, 0.61, 0.69, 1, 1, 1]

𝑘𝑢 = 𝑘𝑣 (due to symmetry)

Example 2

3x10 surface of degree 2 is presented in Figure 5.4(B):

𝑘𝑢 = [0, 0, 0, 1, 1, 1]

𝑘𝑣 = [0, 0, 0, 0.23, 0.30, 0.38, 0.46, 0.53, 0.61, 0.69, 1, 1, 1]

34

The basis functions of NURBS surface are a piece wise polynomial curves calculated

once for the given surface. Basis functions are used for evaluation of any point along the surface

and depends on the knot vector k and control point position.

Control points are initialized as an equally spaced grid along xy-plane. These control

points may be manipulated in order to position the surface tangent to the mesh geometry as

shown in Figure 5.5 where green regions represent the generated surface and red patches display

the surface roughness of the imported mesh.

Figure 5.5 NURBS surface positioned tangent to the surface of an imported mesh.

Designer virtual environment is capable of handling surfaces of 2nd,3rd and 4th degree.

The degree of the surface corresponds to the local surface dependence on position of surrounding

control points and corresponds to the degree of piecewise polynomial curves which compose the

NURBS surface. 2nd degree surfaces are prefered in order to reduce the computational

complexity, while featuring high local dependence of surface geometry with respect to local

35

control point position when compared to higher degree surfaces. This parametrization strategy

allows the use of small surface grids with minimal number of control points. Surfaces of various

degrees are compared in Figure 5.6.

Figure 5.6 NURBS surfaces of degree 2, 3 and 4.

The differentiability properties of NURBS surfaces allows us to obtain the normal vector

along a surface23. Normal and position vectors are sampled across the surface allowing for

toolpath waypoint placement on top of the surface. The pitch of surface sampling may be

adjusted to accommodate the design parameters. Sampled points display normal vectors (yellow

arrows) and provide the user with interactive spheres which can be selected within virtual

environment for waypoint placement. Waypoints are generated on top of the surface by

offsetting the waypoint position by the nozzle diameter in the direction normal to the surface as

shown in Figure 5.7. In addition, the missing geometry patches do not affect the toolpath

placement and allow incomplete geometry to be used for toolpath generation.

36

Figure 5.7 Path generation using NURB surfaces.

37

Chapter 6 Toolpath Designer Application Examples

Several examples of designer virtual environment (VE) usage are presented below.

Designs based on imported geometrical data

Geometrical objects may be imported within the VE in order to produce dimensionally

accurate designs without having to manually measure the object. In this example, the geometry

of a semilunar heart valve was obtained from 3dprint.nih.gov design database. The STL file was

obtained, imported and positioned within the application environment. A cylinder generator was

used four times to build the outer shell of the heart valve as shown in Figure 6.1.

Figure 6.1 Design of semilunar heart valve based on imported STL geometry.

38

Toolpath generation based on 3D scan data of live specimen

Precise deposition of hydrogels in surgical setting can be accomplished by 3D scanning

the specimen and using a convex toolpath generator to produce the toolpaths for the bioprinter to

execute as shown in the following example pictured in Figure 6.2.

Figure 6.2 Testing convex toolpath generator for bioprinting on live specimen.

First, the surgeon exposes the cranium of the specimen and cleans the excess liquid to

prepare the surface for 3D scanning procedure (Figure 6.2A). The exposed cranium is scanned

using a handheld 3D scanner (Figure 6.2B), and the STL geometrical representation of the

surface is imported within designer VE (Figure 6.2C). Toolpaths are generated using the convex

toolpath generator (Figure 6.2D) and are executed by the bioprinter (Figure 6.2E, F).

39

Chapter 7 Conclusions

This thesis has shown how a complete software toolchain for 3DOF and 5DOF

bioprinting platforms can be developed and implemented. This toolchain enables fast prototyping

of bioconstruct design, use of bioprinting platforms in surgical setting, generation of 5DOF

toolpaths on freeform surfaces and enabling users without a technical background to generate

toolpaths for bioprinting applications. This was accomplished by:

• Implementing an interactive 3D toolpath designer virtual environment.

• Development of algorithms which abstract the calculation of waypoint variables

for toolpath creation and manipulation tasks through interactive virtual

environment.

• Development of toolpath generators, which allow fast generation of waypoints

and paths for common shapes.

• Development of bioprinting specific slicer, capable of generating toolpaths within

the concavities of mesh geometry.

• Development of freeform surface generation algorithms in order to bioprint using

5DOF motion stages based on imported mesh data.

40

BIBLIOGRAPHY

1. Au, A. K., Huynh, W., Horowitz, L. F. & Folch, A. 3D-Printed Microfluidics. Angew.

Chemie Int. Ed. 55, 3862–3881 (2016).

2. He, Y. et al. Research on the printability of hydrogels in 3D bioprinting. Nat. Publ. Gr.

(2016). doi:10.1038/srep29977

3. Grand View Research. Grand View Research Updates on the 3D Bioprinting Market.

Entertainment Close - Up (2018).

4. You, F. et al. Application of Extrusion-Based Hydrogel Bioprinting for Cartilage Tissue

Engineering. Int. J. Mol. Sci. 18, 1597 (2017).

5. Derakhshanfar, S. et al. 3D bioprinting for biomedical devices and tissue engineering: A

review of recent trends and advances. Bioact. Mater. 3, 144–156 (2018).

6. Dababneh, A. B. & Ozbolat, I. T. Bioprinting Technology: A Current State-of-the-Art

Review. J. Manuf. Sci. Eng. 136, 061016 (2014).

7. Ozbolat, V. et al. 3D Printing of PDMS Improves Its Mechanical and Cell Adhesion

Properties. ACS Biomater. Sci. Eng. 4, (2018).

8. Povilianskas, A. Development and Implementation of 5 Degree of Freedom Bioprinting

Motion Stage, Control Software and Toolpath Generation Algorithms. BS Honor thesis

(Penn State, 2018).

9. Ranellucci, A. & Lenox, J. Slic3r - About. Available at: https://slic3r.org/about/.

(Accessed: 4th April 2019)

10. Braam, D. Ultimaker Cura: Advanced 3D printing software, made accessible. Available

at: https://ultimaker.com/en/products/ultimaker-cura-software. (Accessed: 4th April 2019)

41

11. Enrique. Skeinforge - RepRap. (2015). Available at: https://reprap.org/wiki/Skeinforge.

(Accessed: 4th April 2019)

12. KISSlicer LLC. KISSlicer. Available at: http://www.kisslicer.com/. (Accessed: 4th April

2019)

13. Gulyas, M., Csiszer, M., Mehes, E. & Czirok, A. Software tools for cell culture-related 3D

printed structures. PLoS One 13, 1–11 (2018).

14. Advanced Solutions. TSIM Software. Available at: https://lifesciences.solutions/tsim/.

(Accessed: 4th April 2019)

15. Kruijff, E., LaViola, J., Poupyrev, I. & Bowman, D. An Introduction to 3D User Interface

Design. Presence Teleoperators Virtual Environ. 10, 96–108 (2001).

16. Biström, J., Cogliati, A. & Rouhiainen, K. Post-WIMP User Interface Model for 3D Web

Applications. 8, (2005).

17. DiPierro, M. The Rise of JavaScript. Comput. Sci. Eng. 20, 9–10 (2018).

18. MrDoob. White paper. gitHub.com (2012).

19. Boyer, P. verb. Available at: http://verbnurbs.com/docs/. (Accessed: 6th April 2019)

20. math.js | an extensive math library for JavaScript and Node.js. Available at:

https://mathjs.org/. (Accessed: 6th April 2019)

21. Piegl, L. Algorithms for Computing Conic Splines. J. Comput. Civ. Eng. 4, 180–198

(1990).

22. Mahmoudi, S. E. et al. Web-based interactive 2D/3D medical image processing and

visualization software. Comput. Methods Programs Biomed. 98, 172–182 (2009).

23. Piegl, L. A. & Tiller, W. The NURBS book. (Springer, 1997).