Download - goto java; (Jfokus)

@MSkarsaune

Martin SkarsauneJava Developer and Co-Owner

A peek into the OpenJDK compiler : goto java;

高馬丁

We’re

Hiring!

@MSkarsaune

GOTO Statements in Java!

@MSkarsaune

GOTO Statement – Objective• Syntax

goto identifier;• Runtime• Program control moves to target statement

• Other Semantics • Target (statement label) must be defined in same

scope, compilation error if not• Potential circular gotos should give compilation

warning.

@MSkarsaune

GOTO Statement – Means• OpenJDK• Open source Java implementation• Javac is implemented in plain Java

• Modify compiler to support GOTO

@MSkarsaune

public class GotoSuccess {

public static void main(String[] args) {one: System.out.print("goto ");two: System.out.print(”J");goto four;three: System.out.print(”2017");goto five;four: System.out.print(”Fokus ");goto three;five: System.out.print("!");

}}

Success Case

@MSkarsaune

GOTOAHEAD

Dijkstra 1968:

@MSkarsaune

What is a Compiler?

Compiler

@MSkarsaune

What is a Compiler?

frontend

backend

@MSkarsaune

Parse Enter Process Attribute Flow Desugar Generat

e

processAnnotations(enterTrees(… parseFiles(…))),…)

…generate(desugar(flow(attribute(…))))

@MSkarsaune


e

• Syntax: goto identifier;• First convert character stream to token stream

(Scanner.java)

GOTO IDENTIFIER SEMI

[g][o][t][o][ ][f][o][u][r][;]

@MSkarsaune


e

• goto is already a reserved word in Java!• Lucky for us, goto is therefore defined as a

TokenKind. • Tokens.java:141: GOTO(“goto”)• The scanner therefore works out of the box!

@MSkarsaune

...import com.sun.tools.javac.parser.Scanner;...


e

final Scanner scanner=factory.newScanner(”goto identifier;”, false);

@MSkarsaune


e


scanner.nextToken();

@MSkarsaune


e


scanner.nextToken();assertThat(”First token is GOTO”, scanner.token().kind, is(GOTO));

@MSkarsaune


e




@MSkarsaune


e



scanner.nextToken();assertThat("Second token is IDENTIFIER", scanner.token().kind,

is(IDENTIFIER));

@MSkarsaune


e




is(IDENTIFIER));


@MSkarsaune


e




is(IDENTIFIER));

scanner.nextToken(); assertThat("Third token is SEMI", scanner.token().kind, is(SEMI));

@MSkarsaune

CompilationUnit

ClassDefinitionFieldDefintion

MethodDefinition

Signature

BodyIfStatement

Goto

visitClassDef(..)

visitMethodDef(..)

visitIf(..)

Abstract Syntax Tree

[g][o][t][o][ ][f][o][u][r][;]

GOTO IDENTIFIER SEMIWikipedia: “the visitor design pattern is a way of separating an algorithm from an

object structure on which it operates”


e

@MSkarsaune

ClassInterface


e

@MSkarsaune

Interface based visitors


e

@MSkarsaune

Class based visitors

public void visitGoto(JCGoto tree) { try { print("goto " + tree.label + ";"); } catch (IOException e) { throw new UncheckedIOException(e); } }

public void visitGoto(JCGoto tree) { //TODO implement}


e

@MSkarsaune

public static class JCLabeledStatement extends JCStatement implements LabeledStatementTree {…public GotoResolver handler;…}

public class GotoResolver { Map<GotoTree, Name> gotos; Map<Name, LabeledStatementTree> targets; List<StatementTree> statementsInSequence; ...}

Helper object


e

@MSkarsaune


e

JavacParser.parseStatement()

case GOTO: { nextToken(); Name label = ident(); JCGoto t = to(F.at(pos).Goto(label, getGotoResolver()); accept(SEMI); return t; }

TreeMaker.java:

public JCGoto Goto(Name label, GotoResolver resolver) { JCGoto tree = new JCGoto(label, resolver); tree.pos = pos; return tree;}

@MSkarsaune

Demo

@MSkarsaune


e


…generate(desugar(flow(attribute(…))))✔

@MSkarsaune


e



@MSkarsaune


e

• Basic sanity testing of compilation unit:

• File name and folder location• Duplicate class names

• Corrections• Add default constructor if no

constructors are declared

@MSkarsaune

Default Constructorpublic class SimpleClass {

}


e

@MSkarsaune

Default Constructorpublic class SimpleClass { public SimpleClass() { super(); }}


e

@MSkarsaune

✔Parse Enter Process Attribut

e Flow Desugar Generate



@MSkarsaune




e

@MSkarsaune

• Process Annitations• Annotation processing API• Part of ordinary javac process since Java

1.6• Plugin API (see javac documentation)


e

@MSkarsaune

• Output controlled by command line switches

@

- proc:only – only process annotations, do not compile- proc:none – do not process annotations


e

@MSkarsaune


e



@MSkarsaune

• Attribution• Semantic checks

–Types–References

• Corrections–Add required calls to super constructor


e

@MSkarsaune

Ensure target label exists in current scopepublic class GotoMissingLabel {

public static void main(String[] args) {one: System.out.print("goto ");two: System.out.print(”Java");goto six;three: System.out.print(”2016");goto five;four: System.out.print(”One ");goto three;five: System.out.print("!");

}}


e

@MSkarsaune


e

Attr.java: @Override public void visitGoto(JCGoto that) {

that.findTarget();if(that.target==null)

log.error(that.pos(), "undef.label", that.label);result = null;

}

class JCGoto: …public void findTarget() { this.target = (JCLabeledStatement)this.handler.findTarget(this);}

@MSkarsaune

Demo

@MSkarsaune


e



@MSkarsaune




e

@MSkarsaune

Flow analysis• Detect unreachable code• Verify assignments• Ensure proper method return• Verify (effectively) final• Check exception flow


e

@MSkarsaune


e

Detect circular gotospublic class GotoCircularWarning {

public static void main(String[] args) { one: System.out.print("goto "); two: System.out.print(”Java"); goto four; three: System.out.print(”2016"); goto five; four: System.out.print(”One "); goto three; five: System.out.println("!"); goto one;//forms infinite loop }}

@MSkarsaune

Flow.FlowAnalyzer class:

compiler.properties:

@Override public void visitGoto(JCGoto tree) { if (tree.handler.detectCircularGotoPosition(tree))

log.warning(tree.pos, "circular.goto"); }

...compiler.warn.circular.goto=circular goto...


e

@MSkarsaune

Demo

@MSkarsaune

✔processAnnotations(enterTrees(…

parseFiles(…))),…)…

generate(desugar(flow(attribute(…))))


e

@MSkarsaune


e



@MSkarsaune

Erase generic types

public class Bridge implements Comparator {

}


e

TransTypes Unlambda Lowerpublic interface Comparator<T> {int compare(T o1, T o1);

}or<T> {

@MSkarsaune

Erase generic types

public class Bridge implements Comparator<Integer> {

public int compare(Integer first, Integer second) {return first - second;

}

}

TransTypes Unlambda Lower


e

public interface Comparator<T> {int compare(T o1, T o1);

} or<T> {

@MSkarsaune

Erasure - Runtime



}

}


e


@MSkarsaune

Erasure - Bridge



} /*synthetic*/ public int compare(Object first, Object second) { return this.compare((Integer)first, (Integer)second); }

}


e


@MSkarsaune


e

TransTypes Unlambda LowerExtract inner classpublic class Outer {

private void foo() { }

public Runnable fooRunner() {return new Runnable() {

public void run() {foo();

}};

}}

@MSkarsaune

Extract inner class


e


public class Outer {private void foo() { }

public Runnable fooRunner() {return new Runnable() {

public void run() {foo();

}};

}}

@MSkarsaune

Extract inner classpublic class Outer {


public Runnable fooRunner() {return new Outer$1(this);

}}


e


class Outer$1 implements Runnable {final Outer this$0;

Outer$1(final Outer this$0) { this.this$0 = this$0; super(); } public void run() {

this$0.foo(); }}

@MSkarsaune

Extract inner class


e


public class Outer {private void foo() { }


}}



this$0.foo(); }}

@MSkarsaune



e

Extract inner classpublic class Outer {



}/*synthetic*/static void access$000(

Outer x0) { x0.foo(); }

}



Outer.access$000(this$0); }}

@MSkarsaune


e

TransTypes Unlambda LowerBoxingList<Integer> list =

Arrays.asList(1, 2);for (Integer i : list) {

System.out.println(”Double: “ + i * 2);}

@MSkarsaune

Boxing


e


List<Integer> list = Arrays.asList(1, 2);

for (Integer i : list) {System.out.println(”Double: “ + i * 2);

}

public static <T> List<T> asList(T... a) { return new ArrayList<>(a); }

@MSkarsaune


e

TransTypes Unlambda LowerBoxingList<Integer> list =

Arrays.asList(Integer.valueOf(1), Integer.valueOf(2));for (Integer i : list) {


@MSkarsaune


e

TransTypes Unlambda LowerUnboxingList<Integer> list =



@MSkarsaune


e

TransTypes Unlambda LowerUnboxingList<Integer> list =


System.out.println(”Double: “ + i.intValue() * 2);}

@MSkarsaune


e

TransTypes Unlambda LowerVarargsList<Integer> list =



public static <T> List<T> asList(T... a) { return new ArrayList<>(a); }

@MSkarsaune


e

TransTypes Unlambda LowerVarargsList<Integer> list =



public static <T> List<T> asList(T... a) { return new ArrayList<>(a);}

@MSkarsaune


e

TransTypes Unlambda LowerVarargs - runtimeList<Integer> list =

Arrays.asList(new Integer[]{Integer.valueOf(1), Integer.valueOf(2)});for (Integer i : list) {


@MSkarsaune


e

TransTypes Unlambda LowerFor each loopList<Integer> list =



@MSkarsaune


e




public interface Iterable<T> { Iterator<T> iterator();}

@MSkarsaune


e


Arrays.asList(new Integer[]{Integer.valueOf(1), Integer.valueOf(2)});for (;;) {


@MSkarsaune



e

For each loopList<Integer> list =

Arrays.asList(new Integer[]{Integer.valueOf(1), Integer.valueOf(2)});for (Iterator i$ = list.iterator();;) {

Integer i System.out.println(”Double: “ + i.intValue() * 2);

}

@MSkarsaune


e


Arrays.asList(new Integer[]{Integer.valueOf(1), Integer.valueOf(2)});for (Iterator i$ = list.iterator(); i$.hasNext();) {

Integer i System.out.println(”Double: “ + i.intValue() * 2);

}

@MSkarsaune



e

For each loopList<Integer> list =

Arrays.asList(new Integer[]{Integer.valueOf(1), Integer.valueOf(2)});for (Iterator i$ = list.iterator(); i$.hasNext();) {

Integer i = (Integer)i$.next(); System.out.println(”Double: “ + i.intValue() * 2);

}

@MSkarsaune


e

TransTypes Unlambda LowerEnumspublic enum Status {

YES, NO, MAYBE}

@MSkarsaune



e

Enums - constructorpublic enum Status {

YES, NO, MAYBEprivate Status(String $enum$name, int $enum$ordinal) {

super($enum$name, $enum$ordinal); }}

public static final Status TRUE = new Status("TRUE", 0);public static final Status FALSE = new Status("FALSE", 1);public static final Status MAYBE = new Status("MAYBE", 2);

@MSkarsaune


e

TransTypes Unlambda LowerEnums - valueOfpublic enum Status {


super($enum$name, $enum$ordinal); } public static Status valueOf(String name) { return (Status)Enum.valueOf(Status.class, name); }}

@MSkarsaune


e

TransTypes Unlambda LowerEnums - valuespublic enum Status {


super($enum$name, $enum$ordinal); } public static Status valueOf(String name) { return (Status)Enum.valueOf(Status.class, name); } private static final Status[] $VALUES = new Status[]{ Status.YES, Status.NO, Status.MAYBE};

public static Status[] values() { return (Status[])$VALUES.clone(); }}

@MSkarsaune


e

TransTypes Unlambda LowerEnum switch statementpublic class SwitchStatus {

void switchStatus(Status status) {

switch (status) { case MAYBE: return;

default: break; } }}

@MSkarsaune



e

Enum switch statementpublic class SwitchStatus {


switch (SwitchStatus$1.$SwitchMap$Status[(status).ordinal()]) { case 1: return;


class SwitchStatus$1 {

}

@MSkarsaune



e

Enum switch statementpublic class SwitchStatus {




class SwitchStatus$1 {static final int[] $SwitchMap$Status = new

int[Status.values().length];

}

@MSkarsaune


e






int[Status.values().length]; [0][0][0]

}

@MSkarsaune


e






int[Status.values().length]; [0][0][1]static { try { SwitchStatus$1.$SwitchMap$Status[Status.MAYBE.ordinal()] = 1; } catch (NoSuchFieldError ex) { } }}

@MSkarsaune


e



@MSkarsaune




e

@MSkarsaune

• Organize initializers• String concatenation• Generate bytecodes


e

@MSkarsaune

• Organize <init> (Constructor)public class InstanceInitialization { String key="key"; String value; public InstanceInitialization(String value) { this.value = value; }

}


e

@MSkarsaune

• Organize <init> (Constructor)public class InstanceInitialization { String key="key"; String value; public InstanceInitialization(String value) {

super(); this.value = value; }

}


e

@MSkarsaune

• Organize <init> (Constructor)public class InstanceInitialization { String key; String value; public InstanceInitialization(String value) {

super(); key = ”key”; this.value = value; }

}


e

@MSkarsaune

• Organize <init> (Constructor)public class InstanceInitialization { String key; String value; public void <init> () {

super(); key = ”key”; this.value = value; }

}


e

@MSkarsaune

• Organize <clinit> (static initialization)public class StaticInitialization { static String key="key"; static { init(); }}


e

@MSkarsaune

• Organize <clinit> (static initialization)public class StaticInitialization { static String key; static { key="key"; init(); }}


e

@MSkarsaune

• Organize <clinit> (static initialization)public class StaticInitialization { static String key; static void <clinit>() { key="key"; init(); }}


e

@MSkarsaune

String concatenationSource code “Generated code”

”string” + value


e

@MSkarsaune


”string” + value new StringBuilder()


e

@MSkarsaune


”string” + value new StringBuilder() .append(”string”)


e

@MSkarsaune


”string” + value new StringBuilder() .append(”string”) .append(value)


e

@MSkarsaune


”string” + value new StringBuilder() .append(”string”) .append(value) .toString()


e

@MSkarsaune

• Goto generation–Luckily for us there is a GOTO byte code–goto <addr>


e

@MSkarsaune

if (<test>) { <ifblock>} else { <elseblock>}<codeafter>

Source code Byte codeCI

9 ifeq<elseblock>

goto<stackmap>

22 <ifblock>

22

<stackmap>29 <codeafter>

29

Java >= 1.6:Stack map frames must be embedded at target of jump instruction

Code generator (Code.java) marked as not alive.Goto instruction added to list of pending jumps (Chain.java)

Pending jumps processed

Normal GOTO usage


e

@MSkarsaune


e

Source code Byte codeCI…label: <somecode>…goto label;

…20

… goto 20

<stackmap><somecode>

Used by goto?Must emit stackmap

Emit goto to label and turn code generation on again

GOTO scenario 1 : jump back

@MSkarsaune

Source code Byte codeCI

GOTO scenario 2 : jump forward…goto label; …label: <somecode>

……

29 <somecode>

goto<stackmap>

Label position not yet known?• emit goto• add to list of pending gotos• turn generation on again

29Label used? • emit stack frame • patch pending

gotos


e

@MSkarsaune

• Goto generation–Gen.java , visitor for code generation–Modify for LabelledStatement–Add implementation for Goto


e

@MSkarsaune

• Gen.java – Labelled Statement

public void visitLabelled(JCLabeledStatement tree) { // if the label is used from gotos, have to emit stack map if (tree.handler.isUsed(tree)) code.emitStackMap(); … }


e

@MSkarsaune

• Gen.java – Goto

public void visitGoto(JCGoto tree) { tree.handler.addBranch(new Chain(code.emitJump(goto_), null, code.state.dup()), tree.target); //normally goto marks code as not alive, turn generation on code.entryPoint();}

Target position known?• Yes – patch immediately• No – add to list of pending gotos


e

@MSkarsaune

Demo

@MSkarsaune

@MSkarsaune

ambdaλ

@MSkarsaune

• Lambda implementation in Java 8–Language change–Compilation–Runtime support

• Many interesting design considerations


e


@MSkarsaune

Simple Examplepublic Comparator<String> lambdaExample() { return (String a, String b) -> a.compareTo(b);}


e


@MSkarsaune

LambdaToMethod.javapublic Comparator<String> lambdaExample() { return (String a, String b) -> a.compareTo(b);}

/*synthetic*/ private static int lambda$lambdaExample$0( , ) { return ;}

final String afinal String b

a.compareTo(b)


e


@MSkarsaune

Runtimepublic Comparator<String> lambdaExample() { return <invokedynamic>LambdaMetafactory.metafactory(}

/*synthetic*/ private static int lambda$lambdaExample$0( final String a, final String b) { return a.compareTo(b);}


e


Lookup(LambdaExample), /*caller*/ "compare", ()Comparator, /*MethodType*/ (Object,Object)int, /*MethodType*/ lambda$lambdaexample$0,/*MethodHandle*/ (String,String)int); /*MethodType*/

final class LambdaExample$$Lambda$1/1834188994 implements Comparator { private LambdaExample$$Lambda$1/1834188994() public int compare(Object,Object)}

public interface Comparator {/*erased*/ int compare(Object o1, Object o2);}

@MSkarsaune

Runtime implementation

LambdaMetaFactory• metaFactory(…)• altMetaFactory(…)

InnerClassLambdaMetafactory

ASM

Lambda Class


e


@MSkarsaune

SerializationLambda Instance

Serialize

SerializedLambda

Deserialize

LambdaMeta

Factory


e


@MSkarsaune

• Possible to back port ?–Capture generated class ?–Compile time generation of inner class!


e


@MSkarsaune

Step 1: Source.java

public boolean allowLambda() { return compareTo( ) >= 0;}

JDK1_8JDK1_5


e


@MSkarsaune

Step 2: Special handling

boolean mustBackportLambda() { return this.target.compareTo(Target.JDK1_8) < 0;}


e


@MSkarsaune

Step 3: Call backport

if(!this.attr.mustBackportLambda()) { result = makeMetafactoryIndyCall(...);}else{ result = new LambdaBackPorter(...).implementLambdaClass(...);}


e


@MSkarsaune

Example implementationprivate static final class Lambda$$2 implements Comparator<String> {

Lambda$$2 { super(); } public int compare(String arg0, String arg1) { return LambdaExample.lambda$lambdaExample$0(arg0, arg1); } public int compare(Object o1, Object o2) { return this.compare((String)o1, (String)o2); }}


e


@MSkarsaune

Example invokingpublic Comparator<String> lambdaExample() { }

return LambdaMetafactory.metafactory(...);return new Lambda$$2();


e


@MSkarsaune

Demo

@MSkarsaune

ambdaλ

@MSkarsaune

@MSkarsaune

Expe

ri

ment

@MSkarsaune


e

• <invokedynamic> “playground”1. @InvokeIndy annotation on method with

reference to boostrap method2. Annotation processor that validates

reference3. Compiler plugin that replaces method

invocation with invokedynamic

@MSkarsaune


e

@IndyMethodpublic class SomeService {

//must refer to a valid public static method //with certain characteristics

@IndyMethod(implementation="no.kantega.jvm.indy.example.SomeProvider", method= "dummyMethod")

public static void doStuff() {throw new UnsupportedOperationException("...");

}

}

@MSkarsaune

no.kantega.jvm.indy.compiler.plugin.IndyAnnotationChecker

Hooking in the processor:META-INF/services/javax.annotation.processing.Processor:


e

@MSkarsaune


e

Annotation processor: setup, compliance + mapping@SupportedAnnotationTypes("no....IndyMethod")

@SupportedSourceVersion(SourceVersion.RELEASE_8)//Java versionpublic class IndyAnnotationChecker extends AbstractProcessor {

//...public boolean process(Set<...> annotations, RoundEnvironment roundEnv) {

for (TypeElement annotation : annotations) {for (Element element : roundEnv.getElementsAnnotatedWith(annotation)) {

//...raise error if earlier than Java 7 or missing pluginIndyMethod indyMethodRef = element.getAnnotation(IndyMethod.class);if (indyMethodRef != null) {

//...check existance of type and compliance of method ...processingEnv.getMessager().printMessage(Kind.ERROR,"...", element);}

}//...

@MSkarsaune


e

...<build>

<plugins><plugin>

<artifactId>maven-compiler-plugin</artifactId><configuration>

<compilerArgument>-proc:none</compilerArgument>

</configuration></plugin>

</plugins></build>

...

Disable annotation processing in the project that implements the plugin (!): pom.xml :

@MSkarsaune


e

• Compiler plugin–Hook straight into the compilation process–Respond to events from compilation process–Make changes to AST

@MSkarsaune


e

no.kantega.jvm.indy.compiler.plugin.IndyPlugin

Hooking in the plugin:META-INF/services/com.sun.source.util.Plugin:

@MSkarsaune


e

public class IndyPlugin implements Plugin {public String getName() {

return "IndyPlugin";}public void init(JavacTask paramJavacTask,

String... paramArrayOfString) {paramJavacTask.addTaskListener(

new GenerateInvokeDynamicHandler());}

}

Typical plugin definition:• Unique name• Delegate to task listener(s)

@MSkarsaune


e

public class GenerateInvokeDynamicHandler implements TaskListener {

public void started(TaskEvent start) {if(start.getKind() == Kind.GENERATE) {

for (Tree tree : start.getCompilationUnit().getTypeDecls()) {tree.accept(new IndyMethodInvocationReplacer(), tree);

}}

}

Task listener:• Receive callback, check stage• Insert visitor to process ASTs

@MSkarsaune


e

public class IndyMethodInvocationReplacer extends TreeScanner {public Object visitMethodInvocation(MethodInvocationTree node, Tree p) {{//...various checks on the method call//...see if annotation processor has created mapping for itMethodSymbol replacementMethod = IndyMethodMappings.getInstance().

mappingFor((MethodSymbol) identifier.sym);if(replacementMethod!= null) {//insert reference to bootstrap

identifier.sym=new Symbol.DynamicMethodSymbol(...); }

}}

}return super.visitMethodInvocation(node, p);

}}

Visitor:• Make modifications

@MSkarsaune


e

<build><plugins>

<plugin><artifactId>maven-compiler-plugin</artifactId><configuration>

<compilerArgs><arg>-Xplugin:IndyPlugin</arg>

</compilerArgs><source>1.8</source><target>1.8</target>

</configuration></plugin>

</plugins></build>

Users:• Enable plugin in compilation

@MSkarsaune

Demo

@MSkarsaune

Wrap Up• The Java compiler is written in pure Java• Compilation is done in phases• Programming language advances (syntactic sugar)

require good compiler support• Lambdas are compiled in a forward compatible manner• Annotation processors and compiler plugins may be

used to tailor the compilation process to specific needs

@MSkarsaune

Resources• OpenJDK

• Source: http://hg.openjdk.java.net/• Compiler hacking tutorial

• http://www.ahristov.com/tutorial/java-compiler.html• Sample code:

• https://github.com/skarsaune/goto• https://github.com/skarsaune/indy-plugin

• Slide pack:• http://www.slideshare.net/MartinSkarsaune

• 60 minute video (Devoxx) :• https://youtu.be/gGTDQq6ZjIk

http://hg.openjdk.java.net/

http://www.ahristov.com/tutorial/java-compiler.html

http://www.ahristov.com/tutorial/java-compiler.html

https://github.com/skarsaune/goto



https://github.com/skarsaune/indy-plugin

http://www.slideshare.net/MartinSkarsaune



https://youtu.be/gGTDQq6ZjIk

https://youtu.be/gGTDQq6ZjIk

@MSkarsaune

Questions or Comments?Martin Skarsaune

Java Developer and Co-Owner

@MSkarsaune

Thank You for Your Time!Martin Skarsaune

Java Developer and Co-Owner