Network of Excellence Benchmark Carlos Oscar Sorzano Techn. Director I 2 PC Natl. Center...
-
Upload
maximillian-rich -
Category
Documents
-
view
215 -
download
0
Transcript of Network of Excellence Benchmark Carlos Oscar Sorzano Techn. Director I 2 PC Natl. Center...
Network of Excellence Benchmark
Carlos Oscar SorzanoTechn. Director I2PC
Natl. Center Biotechnology (CSIC)
NoE Benchmark
NoE Benchmark
• There was a defined task.• The community accepted the challenge.• The result of the challenge could have served
as a starting point to improve 3D angular orientation.
NoE Benchmark• The challenge did not arise from the
community: although the community accepted it.
NoE Benchmark• Evaluation was a black box: the evaluation
Figure of Merit was unknown a priori.
NoE Benchmark• There was not a clear interchange standard:
there was no way of testing that the submitted results where actually in the correct format, coordinate system, Euler signs, …
NoE Benchmark• There was a single dataset: there was no
exploration of different packages performing better under different circumstances.
NoE Benchmark• Egos were too high
NoE Benchmark• It was a single shot.
1891: 10.8s1906: 10.6s1911: 10.5s1924: 10.4s1930: 10.3s1936: 10.2s…2009: 9.58s
For future benchmarks• Well defined task.• Well defined interchange format.• Task proposed by the community.• Different datasets.• Transparent evaluation with pre-test
datasets.• Acceptance of competition as a healthy way
to improve.• Possibility to withdraw from competition.• Competitions should be repeated over time.