Scala Next

SF Scala meetup Dec 8th, 2011

*Scala Today

*Some adoption vectors:Web platformsTrading platformsFinancial modelingSimulationFast to first product, scalable afterwards

Github vs. Stack OverflowRedMonk: Revisiting the Dataists Programming Language Rankings*Typesafe Confidential

Typesafe Confidential

Commercial AdoptionScala jobs tripled in last yearNow at estimated 100,000 developers*Typesafe Confidential

Typesafe Confidential

*Scala 2.8:

(Only 17 months ago!)

New collectionsPackage objectsContext boundsBetter implicits...

*Scala 2.9:Parallel collectionsDelayedInit and AppFaster REPLProgress on IDEs: Eclipse, IntelliJ, Neatbeans, ENSIMEBetter docsLots of bug fixes

Parallel CollectionsUse Java 7 Fork Join frameworkSplit work by number of ProcessorsEach Thread has a work queue that is split exponentially. Largest on end of queueGranularity balance against scheduling overheadOn completion threads work steals from end of other thread queues

*

*... and its usageimport java.util.ArrayList;...Person[] people;Person[] minors;Person[] adults;{ ArrayList minorsList = new ArrayList(); ArrayList adultsList = new ArrayList(); for (int i = 0; i < people.length; i++) (people[i].age < 18 ? minorsList : adultsList) .add(people[i]); minors = minorsList.toArray(people); adults = adultsList.toArray(people);} ... in Java:... in Scala:val people: Array[Person]val (minors, adults) = people partition (_.age < 18)A simple pattern matchAn infix method callA function value

*Going Parallel ?... in Java:... in Scala:val people: Array[Person]val (minors, adults) = people.par partition (_.age < 18)

General Collection HierarchyGenTraversableGenIterableGenSeqTraversableIterableSeqParIterableParSeq*Remove this layer in 2.10?

Going DistributedCan we get the power of parallel collections to work on 10000s of computers?Hot technologies: MapReduce (Googles and Hadoop)But not everything is easy to fit into that moldSometimes 100s of map-reduce steps are needed.Distributed collections retain most operations, provide a powerful frontend for MapReduce computations.Scalas uniform collection model is designed to also accommodate parallel and distributed.Projects at Google (Cascade), Berkeley (Spark), EPFL.*

*Scala next:Eclipse IDE

Play web framework 2.0

Akka 2.0

Scala 2.10

*Scala Eclipse IDENow in RC2Final expected before the end of the year.

Goalsreliable (no crashes/lock ups)responsive (never wait when typing)work with large projects/filesScala compiler (80k LOC), 4-5000 LOC/fileadvanced use of the type system: path-dependent types, self-types, mix-ins

FeaturesKeep it simplehighlight errors as you typecompletions (including implicits)hyperlinkingproject builder (+ dependent projects)Support mixed Java-Scala projectsall features should work between Java/Scala sourcesJUnit Test Runner should pick up testsMore stuff based on external libraries(some) refactoring, code formatter, mark occurrences, structured selections, show inferred semi-colons

Features (3)based on external libraries(some) refactoring code formattermark occurrencesstructured selectionsshow inferred semi-colons

@jonifreemanJoni Freeman

Latest Scala Eclipse plugin works surprisingly well! Even manages our mixed Java/Scala project. Kudos to the team! #scala

@esorribasEduardo Sorribas

The latest beta of the Scala IDE for eclipse is much better. I'm starting to like it.

@jannehietamakiJanne Hietamki

After years of misery, the Eclipse Scala plugin actually seems to work quite well.

ArchitectureUse the full-blown Scala compiler for:interactive error highlight, completion, hyperlinkingturning Scala symbols into Java model elements

Weave the JDT compiler when it needs helpJDT was NOT meant to be extended

Why rely on scalac?reuse (type-checker == 1-2 person years)consistencycompiler pluginsWhy not?SPEED(very) tight dependency on the Scala version

Presentation Compilerasynchronousinterruptibletargetedstop after type-checking

Result is communicated through a SyncVar

All compiler activity happens on PC threadcompile loaded files when work queue is empty (in the background)Check work queue when type checker reaches safe-points in the ASTDrop everything when a file is changed (AskReload)

*Implementation

1 type-checker run / instance --> 100s of type-check runs / minutememory leaksside-effects/stateout-of-order and targeted type-checking

needed to improve the compiler2.9.x, 2.10 (trunk)what about 2.8?2.8.2, 2.8.3-SNAPSHOT

New: Play Framework 2.0Play Framework is an open source web application framework, inspired by Ruby on Rails, for Java and ScalaPlay Framework 2.0 retains full Java support while moving to a Scala core and builds on key pieces of the Typesafe Stack, including Akka middleware and SBTPlay will be integrated in TypeSafe stack 2.0Typesafe will contribute to development and provide commercial support and maintenance.

RoadmapMay 2011Oct 2011Q1 2012Q3 2012Scala 2.9.0Akka 1.1Scala 2.9.1Akka 1.2Scala 2.9.xAkka 2.0Play 2.0Scala 2.10Akka 2.xPlay 2.xSlick (DB)

*Scala 2.10:New reflection frameworkReificationtype DynamicMore IDE improvements: find-references, debugger, worksheet.Faster buildsSIPs: string interpolation, simpler implicits.

ETA: Early 2012.

New in Scala 2.10: DynamicType Dynamic bridges the gap between static and dynamic typing.Method calls get translated to applyDynamicGreat for interfacing with dynamic languages (e.g. JavaScript)* class JS extends Dynamic { def applyDynamic(methName: String, args: Any*): Any = { println("apply dynamic "+methName+args.mkString("(", ",", ")")) } } val x = new JS x.foo(1) // x.applyDynamic(foo, 1) x.bar // x.applyDynamic(bar)

Proposed for Scala 2.10: SIP 11: String interpolationIdea: Instead of Bob is + n + years oldwrite:sBob is $n years oldwhich gets translated tonew StringContext(Bob is, years old).s(n)Here, s is a library-defined method for string interpolation.

*

This can be generalized to other string processors besides s:xml ${linktext}

scala scala.concurrent.transaction.withinTransaction { (implicit currentTransaction: Transaction) =>$expr }

*

Proposed for Scala 2.10: SIP 12: Uncluttering controlShould be able to write:

if x < 0 then x else x

while x > 0 do { println(x); x -= 1 }

for x

Proposed for Scala 2.10: SIP 13: Implicit classesVariation: Add @inline to class def to get speed of extension methods.*

New in Scala 2.10: ReflectionPreviously: Needed to use Java reflection,no runtime info available on Scalas types.

Now you can do:

*

(Bare-Bones) Reflection in Java*Want to know whether type A conforms to B?Write your own Java compiler!Why not add some meaningful operations?

Need to write essential parts of a compiler (hard).

Need to ensure that both compilers agree (almost impossible).

How to do Better?Problem is managing dependencies between compiler and reflection.Time to look at DI again.*Dependency InjectionIdea: Avoid hard dependencies to specific classes.Instead of calling specific classes with new, have someone else do the wiring.

Using Guice for Dependency Injection*(Example by Jan Kriesten)

... plus some Boilerplate*

Dependency Injection in Scala*Components are classes or traitsRequirements are abstract valuesWiring by implementing requirement valuesBut what about cyclic dependencies?

The Cake Pattern*Requirements are types of thisComponents are traitsWiring by mixin composition

Cake Pattern in the CompilerThe Scala compiler uses the cake pattern for everythingHeres a schema:

(In reality there are about ~20 slices in the cake.)*

Towards Better ReflectionCan we unify the core parts of the compiler and reflection?

Compiler Reflection

Different requirements: Error diagnostics, file access, classpath handling - but we are close!

*

Compiler Architecture*reflect.internal.Universensc.Global (scalac)reflect.runtime.MirrorProblem: This exposes way too much detail!

Complete Reflection Architecture Cleaned-up facade:

Full implementation:*reflect.internal.Universensc.Global (scalac)reflect.runtime.Mirrorreflect.api.Universe / reflect.mirror

How to Make a Facade*The FacadeThe ImplementationInterfaces are not enough!

ConclusionScala is a very regular language when it comes to composition: Everything can be nested:classes, methods, objects, typesEverything can be abstract:methods, values, typesThe type of this can be declared freely, can thus express dependencies This gives great flexibility for SW architecture, allows us to attack previously unsolvable problems.

*

Going further: Parallel DSLsMid term, research project: How do we keep tomorrows computers loaded?How to find and deal with 10000+ threads in an application?Parallel collections and actors are necessary but not sufficient for this.Our bet for the mid term future: parallel embedded DSLs.Find parallelism in domains: physics simulation, machine learning, statistics, ...Joint work with Kunle Olukuton, Pat Hanrahan @ Stanford.EPFL side funded by ERC.

*

EPFL / Stanford ResearchApplicationsDomainSpecificLanguagesHeterogeneousHardwareDSLInfrastructureOOO CoresSIMD CoresThreaded CoresSpecialized CoresProgrammableHierarchiesScalable CoherenceIsolation & AtomicityOn-chipNetworksPervasive Monitoring*

Example: Liszt - A DSL for Physics SimulationMesh-basedNumeric SimulationHuge domains millions of cellsExample: Unstructured Reynolds-averaged Navier Stokes (RANS) solver

*

Liszt as Virtualized Scalaval // calculating scalar convection (Liszt)

val Flux = new Field[Cell,Float]val Phi = new Field[Cell,Float]val cell_volume = new Field[Cell,Float]val deltat = .001...untilconverged { for(f

*Follow us on twitter: @typesafescala-lang.org

typesafe.comakka.ioscala-lang.org

Crossed over 1% of Java jobs. If Java has 10 million developers, Scala has 100,000.**This leads to our vision, applications driven by a set of interoperable DSLs. We are developing DSLs to provide evidence as to their effectiveness in extracting parallel performance. But we are also very interested in empowering other to easily build such DSLs, so we are investing heavily in developing frameworks and runtimes to make parallel DSL development easier. And the goal is to run single source programs on a variety of very different hardware targets.*Liszt is another language we have implemented. It is designed to support the creation of solvers for mesh-based partial differential equations. Problems in this domain typically simulate complex physical systems such as fluid flow or mechanics by breaking up space into discrete cells. A typical mesh may contain hundreds of millions of these cells (here we are visualizing a scram-jet designed to work at hypersonic speeds). Liszt is an ideal candidate for a DSL because while the problems are large and highly parallel, the mesh introduces many data-dependencies that are difficult to reason about, making writing solvers tedious.

**