Post on 15-Sep-2020
Revisiting Sparse Dynamic
Programming for the 0/1 Knapsack
Problem
Tarequl Islam Sifat
Tarequl.Sifat@colostate.edu
Corespeq Inc
Fort Collins, Colorado, USA
Nirmal Prajapati
prajapati@lanl.gov
Los Alamos National Laboratory
Los Alamos, New Mexico, USA
Sanjay Rajopadhye
Sanjay.Rajopadhye@colostate.edu
Department of Computer Science
Colorado State University
Fort Collins, Colorado, USA
0/1 Knapsack Problem Statement
◼ Given a set of 𝑁 items numbered from 1 up to 𝑁, each with a weight
𝑤𝑖 and a profit 𝑝𝑖, along with maximum capacity 𝐶, we must
𝑚𝑎𝑥𝑖𝑚𝑖𝑧𝑒
𝑖=1
𝑁+1
𝑝𝑖𝑥𝑖
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜
𝑖=1
𝑁+1
𝑤𝑖𝑥𝑖 < 𝐶 𝑎𝑛𝑑 𝑥𝑖 ∈ {0,1}
2
Sparse DP Algorithm for solving 0/1-
Knapsack Problem◼ A “sparse” KPDP algorithm (SKPDP) has been known for a while.
◼ Conventional KPDP algorithm generates a DP table that contains
many repeated values.
◼ SKPDP does not calculate the repeated values in the DP table.
◼ So far there has been no quantitative analysis of its benefits.
3
Contributions
◼ Quantitative analysis of Sequential SKPDP
◼ Exploration of two parallelization techniques for SKPDP and their
performance analysis
◼ Comparison of SKPDP with Branch-and-Bound
4
Problem Instance Generation
For Quantitative Analysis◼ Uncorrelated : 𝑝𝑖 = 𝑟𝑎𝑛𝑑𝑜𝑚(𝑟𝑎𝑛𝑔𝑒)
◼ Weakly Correlated : 𝑝𝑖 = 𝛼𝑤𝑖 + 𝑟𝑎𝑛𝑑𝑜𝑚(𝑟𝑎𝑛𝑔𝑒)
◼ Strongly Correlated: 𝑝𝑖 = 𝛼𝑤𝑖 + 𝛽 [Hardest Problem Instances]
◼ Subset Sum: 𝑝𝑖 = 𝑤𝑖
where 𝛼, 𝛽 are constants, 𝑝𝑖 is profit and 𝑤𝑖 is weight of each item.
5
Punch Line
◼ For KP instances with significantly large capacity than the number of
items (C >> N)
◼ If the problem instance is weakly correlated, the operation count of
SKPDP algorithm is invariant with respect to capacity (𝐶).
◼ If the problem instance is strongly correlated, the operation count of
SKPDP algorithm is exponentially less than KPDP.
6
Punch Line
7
Weakly Correlated Instances Strongly Correlated Instances
Dynamic Programming Solution
8
for ( k=1 ; k < N ; k++ ) {for ( c=0 ; c <= C ; c++ ) {
if( c < weights[k] )M[k,c] = M[k-1,c];
elseM[k,c]=MAX(M[k-1,c], M[k-1,c-weights[k]]+profits[k]);
}}
𝑀 𝑘, 𝑐 = ቊ𝑀 𝑘 − 1, 𝑐 𝑖𝑓 𝑤𝑘 > 𝑐
max 𝑀 𝑘 − 1, 𝑐 ,𝑀 𝑘 − 1, 𝑐 − 𝑤𝑘 + 𝑝𝑘 𝑒𝑙𝑠𝑒
Example Problem Instance
Item No. Profits Weights
1 1 1
2 6 2
3 18 5
4 22 6
5 28 7
9
𝑁 = 5 𝐶 = 11
Dynamic Programming Table
10
𝑀 𝑘, 𝑐 = ቊ𝑀 𝑘 − 1, 𝑐 𝑖𝑓 𝑤𝑘 > 𝑐
max 𝑀 𝑘 − 1, 𝑐 ,𝑀 𝑘 − 1, 𝑐 − 𝑤𝑘 + 𝑝𝑘 𝑒𝑙𝑠𝑒
Capacity
Item# (weight,profit) 0 1 2 3 4 5 6 7 8 9 10 11
0 0 0 0 0 0 0 0 0 0 0 0 0
1 (1,1) 0 1 1 1 1 1 1 1 1 1 1 1
2 (2,6) 0 1 6 7 7 7 7 7 7 7 7 7
3 (5,18) 0 1 6 7 7 18 19 24 25 25 25 25
4 (6,22) 0 1 6 7 7 18 22 24 28 29 29 40
5 (7,28) 0 1 6 7 7 18 22 28 29 34 35 40
c
k
Memory Efficient KPDP
◼ Only need the current row of the DP table to calculate the next
row
◼ The whole table does not have to be stored
◼ This way we can find the optimal profit value
◼ We can find the exact solution including which items are taken in
the optimal solution by using a divide-and-conquer strategy
◼ Divide-and-Conquer strategy doubles the number of
computations, 2𝑁𝐶
◼ Reduces memory requirement by a factor of 𝑁/2
11
“Sparsity” in the current context
12
“Sparsity” in the current context
Capacity
Item# (weight,profit) 0 1 2 3 4 5 6 7 8 9 10 11
0 0 0 0 0 0 0 0 0 0 0 0 0
1 (1,1) 0 1 1 1 1 1 1 1 1 1 1 1
2 (2,6) 0 1 6 7 7 7 7 7 7 7 7 7
3 (5,18) 0 1 6 7 7 18 19 24 25 25 25 25
4 (6,22) 0 1 6 7 7 18 22 24 28 29 29 40
5 (7,28) 0 1 6 7 7 18 22 28 29 34 35 40
13
Item#
0 <0,0>
1 (1,1) <0,0>,<1,1>
2 (2,6) <0,0>,<1,1>,<2,6>,<3,7>
3 (5,18) <0,0>,<1,1>,<2,6>,<3,7>,<5,18>,<6,19>,<7,24>,<8,25>
4 (6,22) <0,0>,<1,1>,<2,6>,<3,7>,<5,18>,<6,22>,<7,24>,<8,28>,<9,29>,<11,40>
5 (7,28) <0,0>,<1,1>,<2,6>,<3,7>,<5,18>,<6,22>,<7,28>,<8,29>,<9,34>,<10,35>,<11,40>
Building the Sparse Table
Add-Merge-Kill
14
{1,2,3} <0,0>, <1,1>, <2,6>, <3,7>, <5,18>, <6,19>, <7,24>, <8,25>
{1,2,3,4} <0,0>, <1,1>, <2,6>, <3,7>, <5,18>, <6,22>, <7,24>, <8,28>, <9,29>, <11,40>
When we include the 4th item (Weight: 6, Profit: 22) in our choice
<0,0>,<1,1>,<2,6>,<3,7>,<5,18>,<6,19>,<7,24>,<8,25>
Building the Sparse Table
Add-Merge-Kill
14
{1,2,3} <0,0>, <1,1>, <2,6>, <3,7>, <5,18>, <6,19>, <7,24>, <8,25>
{1,2,3,4} <0,0>, <1,1>, <2,6>, <3,7>, <5,18>, <6,22>, <7,24>, <8,28>, <9,29>, <11,40>
When we include the 4th item (Weight: 6, Profit: 22) in our choice
<0,0>,<1,1>,<2,6>,<3,7>,<5,18>,<6,19>,<7,24>,<8,25>
Building the Sparse Table
Add-Merge-Kill
14
{1,2,3} <0,0>, <1,1>, <2,6>, <3,7>, <5,18>, <6,19>, <7,24>, <8,25>
{1,2,3,4} <0,0>, <1,1>, <2,6>, <3,7>, <5,18>, <6,22>, <7,24>, <8,28>, <9,29>, <11,40>
When we include the 4th item (Weight: 6, Profit: 22) in our choice
Add (6,22) to each pair,
<6,22>,<7,23>,<8,28>,<9,29>,<11,40>,<12,41>,<13,46>,<14,47>
<0,0>,<1,1>,<2,6>,<3,7>,<5,18>,<6,19>,<7,24>,<8,25>
Building the Sparse Table
Add-Merge-Kill
14
{1,2,3} <0,0>, <1,1>, <2,6>, <3,7>, <5,18>, <6,19>, <7,24>, <8,25>
{1,2,3,4} <0,0>, <1,1>, <2,6>, <3,7>, <5,18>, <6,22>, <7,24>, <8,28>, <9,29>, <11,40>
When we include the 4th item (Weight: 6, Profit: 22) in our choice
Add (6,22) to each pair,
<6,22>,<7,23>,<8,28>,<9,29>,<11,40>
<0,0>,<1,1>,<2,6>,<3,7>,<5,18>,<6,19>,<7,24>,<8,25>
Building the Sparse Table
Add-Merge-Kill
14
{1,2,3} <0,0>, <1,1>, <2,6>, <3,7>, <5,18>, <6,19>, <7,24>, <8,25>
{1,2,3,4} <0,0>, <1,1>, <2,6>, <3,7>, <5,18>, <6,22>, <7,24>, <8,28>, <9,29>, <11,40>
When we include the 4th item (Weight: 6, Profit: 22) in our choice
<6,22>,<7,23>,<8,28>,<9,29>,<11,40>
<0,0>,<1,1>,<2,6>,<3,7>,<5,18>,<6,19>,<7,24>,<8,25>
Building the Sparse Table
Add-Merge-Kill
14
{1,2,3} <0,0>, <1,1>, <2,6>, <3,7>, <5,18>, <6,19>, <7,24>, <8,25>
{1,2,3,4} <0,0>, <1,1>, <2,6>, <3,7>, <5,18>, <6,22>, <7,24>, <8,28>, <9,29>, <11,40>
When we include the 4th item (Weight: 6, Profit: 22) in our choice
<6,22>,<7,23>,<8,28>,<9,29>,<11,40>
Building the Sparse Table
Add-Merge-Kill
15
Generation of Problem Instances
◼ The fraction of objects that can fit in the knapsack on average, 1/𝜆
𝑊𝑎𝑣𝑔 =𝜆𝐶
𝑁
◼ The set of weights, 𝑤𝑖 is generated with a normal distribution that
has a mean of 𝑊𝑎𝑣𝑔
◼ For weakly correlated problem instances, the correlation between
the weights and profits is controlled by a noise factor, 𝜎𝑝𝑖 = 𝛼𝑤𝑖 + 𝑅𝑎𝑛𝑑𝑜𝑚 𝐼𝑛𝑡𝑒𝑔𝑒𝑟 𝑏𝑒𝑡𝑤𝑒𝑒𝑛 [−𝜎𝑊𝑎𝑣𝑔, 𝜎𝑊𝑎𝑣𝑔]
◼ For strongly correlated problem instances 𝜎 is irrelevant,
𝑝𝑖 = 𝛼𝑤𝑖 + 𝛽
16
Gain
𝐺𝑎𝑖𝑛 = 1 −𝐼𝑡𝑒𝑟𝑎𝑡𝑖𝑜𝑛𝑠 𝑖𝑛 𝑆𝐾𝑃𝐷𝑃
𝐼𝑡𝑒𝑟𝑎𝑡𝑖𝑜𝑛𝑠 𝑖𝑛 𝐾𝑃𝐷𝑃 (= 2𝑁𝐶)
The range of Gain is (-1, 1)
◼ A value of gain close to 1 means that the number of iterations
in SKPDP is insignificant compared to KPDP
◼ A value of gain close to -1 mean that we have the worst-case
scenario for SKPDP.
17
Gain
18
𝜆 = 2, 𝜎 = 0.1%
SKPDP vs KPDP
19
Weakly Correlated Instances Strongly Correlated Instances
𝜆 = 2, 𝜎 = 0.1% 𝑁 = 256, 𝜆 = 8, 𝜎 = 0.1%
Impact of 𝜎 on the sparsity
20
𝑝 = 𝛼𝑤 + 𝑅𝑎𝑛𝑑𝑜𝑚 𝐼𝑛𝑡. 𝐼𝑛 −𝜎𝑊𝑎𝑣𝑔, 𝜎𝑊𝑎𝑣𝑔 ; 𝜆 = 2
Impact of 𝜆 on the sparsity
21
𝑊𝑎𝑣𝑔 =𝜆𝐶
𝑁; 𝜎 = 0.1%
Impact of 𝜎 and 𝜆 on the sparsity
22
𝑁 = 210, 212 < C < 250
Parallelization of SKPDP
◼ Fine-Grained Parallelization
◼ Coarse-Grained Parallelization
23
Fine-Grained Parallelization
24
Fine-Grained Parallelization
Item#
0 <0,0>
1 (1,1) <0,0>,<1,1>
2 (2,6) <0,0>,<1,1>,<2,6>,<3,7>
3 (5,18) <0,0>,<1,1>,<2,6>,<3,7>,<5,18>,<6,19>,<7,24>,<8,25>
4 (6,22) <0,0>,<1,1>,<2,6>,<3,7>,<5,18>,<6,22>,<7,24>,<8,28>,<9,29>,<11,40>
5 (7,28) <0,0>,<1,1>,<2,6>,<3,7>,<5,18>,<6,22>,<7,28>,<8,29>,<9,34>,<10,35>,<11,40>
25
{1,2,3} <0,0>, <1,1>, <2,6>, <3,7>, <5,18>, <6,19>, <7,24>, <8,25>
{1,2,3,4} <0,0>, <1,1>, <2,6>, <3,7>, <5,18>, <6,22>, <7,24>, <8,28>, <9,29>, <11,40>
When we include the 4th item (Weight: 6, Profit: 22) in our choice
Remember the example used to demonstrate Add-Merge-Kill?
Fine-Grained Parallelization
26
<0,0>,<1,1>,<2,6>,<3,7>,<5,18>,<6,19>,<7,24>,<8,25>4th row:
Fine-Grained Parallelization
26
<0,0>,<1,1>,<2,6>,<3,7>,<5,18>,<6,19>,<7,24>,<8,25>4th row:
<5,18>,<6,19>,<7,24>,<8,25><0,0>,<1,1>,<2,6>,<3,7>
Fine-Grained Parallelization
26
<0,0>,<1,1>,<2,6>,<3,7>,<5,18>,<6,19>,<7,24>,<8,25>4th row:
<5,18>,<6,19>,<7,24>,<8,25><0,0>,<1,1>,<2,6>,<3,7>
<0,0>,<1,1>,<2,6>,<3,7>,<6,22>,<7,23>,<8,28>,<9,29>
<5,18>,<6,19>,<7,24>,<8,25>,<11,40>
Fine-Grained Parallelization
26
<0,0>,<1,1>,<2,6>,<3,7>,<5,18>,<6,19>,<7,24>,<8,25>4th row:
<5,18>,<6,19>,<7,24>,<8,25><0,0>,<1,1>,<2,6>,<3,7>
<0,0>,<1,1>,<2,6>,<3,7>,<6,22>,<7,23>,<8,28>,<9,29>
<5,18>,<6,19>,<7,24>,<8,25>,<11,40>
At most 𝑤𝑖 pairs from the suffix of the preceding part
Fine-Grained Parallelization
26
<0,0>,<1,1>,<2,6>,<3,7>,<5,18>,<6,19>,<7,24>,<8,25>4th row:
<5,18>,<6,19>,<7,24>,<8,25><0,0>,<1,1>,<2,6>,<3,7>
<0,0>,<1,1>,<2,6>,<3,7>,<6,22>,<7,23>,<8,28>,<9,29>
<5,18>,<6,19>,<7,24>,<8,25>,<11,40>
At most 𝑤𝑖 pairs from the suffix of the preceding part
At most 𝑤𝑖 pairs from the prefix of the following part
Fine-Grained Parallelization
26
<0,0>,<1,1>,<2,6>,<3,7>,<5,18>,<6,19>,<7,24>,<8,25>4th row:
<5,18>,<6,19>,<7,24>,<8,25><0,0>,<1,1>,<2,6>,<3,7>
<0,0>,<1,1>,<2,6>,<3,7>,<6,22>,<7,23>,<8,28>,<9,29>
<5,18>,<6,19>,<7,24>,<8,25>,<11,40>
At most 𝑤𝑖 pairs from the suffix of the preceding part
At most 𝑤𝑖 pairs from the prefix of the following part
In the worst-case extra merge-kill we must do at each point of stitching is 2𝑤𝑖
Fine-Grained Parallelization
26
<0,0>,<1,1>,<2,6>,<3,7>,<5,18>,<6,19>,<7,24>,<8,25>4th row:
<5,18>,<6,19>,<7,24>,<8,25><0,0>,<1,1>,<2,6>,<3,7>
<0,0>,<1,1>,<2,6>,<3,7>,<6,22>,<7,23>,<8,28>,<9,29>
<5,18>,<6,19>,<7,24>,<8,25>,<11,40>
<0,0>,<1,1>,<2,6>,<3,7>,<5,18>,<6,22>,<7,24>,<8,28>,<9,29>,<11,40>
Coarse-Grained Parallelization
27
Coarse-Grained Parallelization
◼ The computation occurs in batches of P rows.
◼ Every 𝑃𝑡ℎ row of the DP table is stored (completely) in the
memory.
◼ Other 𝑃 − 1 rows only stores a local buffer of size 𝑤𝑖
28
<0,0>,<1,1>,<2,6>,<3,7>,<5,18>,<6,19>,<7,24>,<8,25>4rd row:
k
j
At most 𝑤𝑖
Coarse-Grained Parallelization
◼ The local buffers are basically a FIFO stack and implemented
with a rotating array.
◼ These rotating arrays have a length of 𝑊𝑚𝑎𝑥 = max(𝑤𝑖)
◼ Extra memory required by the implementation is 𝑃 ∗𝑊𝑚𝑎𝑥
29
Performance of Coarse-Grained
Parallelization
30
𝜆 = 8, 𝜎 = 0.1%
SKPDP vs BnB
31
Future work
◼ A model to predict the performance gain by SKPDP
◼ Ordering the items by weight values to maximize sparsity
◼ Choosing the best input parameters for the coarse-grained
parallelization of SKPDP
32
Conclusion
◼ SKPDP shows exponential performance improvement using
SKPDP over KPDP.
◼ Coarse-grained parallelization shows promise.
◼ For most problem instances BnB with proper heuristics does
better than SKPDP.
◼ For some hard problem instances SKPDP performs better
than BnB.
33
34
Thank you!