Skip to main content

Deprecate JAVAC

27 replies [Last post]
brucechapman
Offline
Joined: 2004-03-18
Points: 0

-- wait for laughter to die down --

I am serious.

-- wait for laughter to die down - again --

let me explain (to anyone who hasn't discounted me as a complete nutter yet, and is still reading)

5.0 has a new tool "apt" which is a better javac. It allows us (mere mortals) to run our own code inside the compiler which can ...
* examine the source being compiled (via mirror API - similar to reflection or javadoc API)
* generate other code (sourcecode which will then be compiled, as well as bytecode if you have the inclination),
* and generate compiler errors if we detect something out of spec.

Do you want a compiler error when you implement an interface and the javadocs say that implementations must have a public default constructor, and yours doesn't?

Well "apt" could do that - (in fact I've coded it already - and it works - I'll probably put the stuff up as a java.net project when its a bit more stable and I have a few more useful things included, and a bit of time and ... ).

Now for sure, apt does need a little more work (check the bug parade keyword "apt"), but as far as teenagers go, its pretty damn useful.

It could also do with a bit more work under the hood in terms of reusing information from the apt processing phase in the compilation phase, but thats just a performance thing (and might have already been done because I can't find the bug anymore that suggested this).

Apt's one weakness is that you can't force people to use it, so whatever useful stuff you do with it, others can just bypass it and use javac instead, and all those runtime errors you would have caught at compile time, slip through again. Argh.

Hence - deprecate javac, but also rename apt to be javac (or just put the apt processing functionality in javac - but that doesn't make such a controversial subject heading does it :) ).

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
rreyelts
Offline
Joined: 2003-06-10
Points: 0

Oh and one other thing. Don't let me rain on your rapt parade. There may be a lot of neat things that you can do with apt, while the problems I'm trying to solve just require more. In any case, I'll try to keep abreast of your accomplishments there.

God bless,
-Toby Reyelts

rreyelts
Offline
Joined: 2003-06-10
Points: 0

> 1. What does the generated code look like? and how do you call the generated code?

To be honest, I'm still working that out. [b]I really, really, really wish APT gave you a modifiable AST, which you could then send through another compiler pass.[/b] It would simplify so many annotation processing tasks.

Right now, I'm considering using JavaCC to parse and generate a new version of the source, with the contract code inlined. The main drawback there is out-of-synch line numbers. To get around that, I'm considering writing some sort of preprocessor that can read #line directives, strip them from a file, run the file through javac, then update the line number tables of the generated bytecode. What a pain in the ass.

The other approach would be to just munge the bytecode, but that has very practical problems in this case. For example, once the classfile is compiled, I have no idea what the import statements were, so it's impossible for me to correctly parse the Java expression in the annotation.

I'm also considering a "dual-strategy" approach, where I would run an annotation processor that walks the AST for source-code only information I'm looking for (import statements, etc...), which would then feed it to a bytecode processor.

> If you don't have another "home"

Thanks for the offer, but I don't think of this as an APT plugin. APT sucks when you need to do actual modification of the class (unless I'm really missing something). For example, I saw your LongString annotation, and I think it's really goofy that you have to call a method to initialize your string. But that's the only strategy that makes sense with APT, because it allows you to generate code outside of the class, as opposed to modifying the class itself.

> rapt may still be of interest as a place to discuss apt issues.

I'll keep that in mind. It's too bad APT is so limited, right now.

God bless,
-Toby Reyelts

brucechapman
Offline
Joined: 2004-03-18
Points: 0

>
> Thanks for the offer, but I don't think of this as an
> APT plugin. APT sucks when you need to do actual
> modification of the class (unless I'm really missing
> something). For example, I saw your LongString
> annotation, and I think it's really goofy that you
> have to call a method to initialize your string. But
> that's the only strategy that makes sense with APT,
> because it allows you to generate code outside of the
> class, as opposed to modifying the class itself.

Hi,

True, but first, we can be really thankful that we are actually able to run our code inside the compiler. Sure there are limits to what we can do, but we haven't even started to explore the possibilities much. What has been done so far doesn't go too far out on a limb that might be regretted later.

Think of this the other way around, would you be happy using an annotation if there was a possibility that some Apt processor, hiding in a jar file somewhere on your classpath could modify the ast of your code without your knowledge. In fact, as apt is at the moment you don't even need an annotation for a processor to run. Quite frankly that is scary enough for me that I am more than happy to live with the current limitations.

What I want to do is explore the possibilities. LongString was the first (if you discount that other project which you know I am working on). Some of these will (hopefully) be really cool with little syntactic baggage. E.G. I can live with the static method call in LongStrings because that is heaps better than quoting multiple String literals and concatenating them.

Some of the things I would like to try might just turn out too ugly to use. However I feel we need to go there and find out what works and what doesn't.

The DBC stuff I had sketched out was going to work a bit like LongString in that there would be a (say) Check class per package with a method with the same name and args as the annotated method, but with an additonal 1st parameter of same type as the enclosing class so you can pass in "this". I thought I would make the methods return true always, so you can do this
[code]
/**
* Returns the sum of Math.abs( value[ i ] ) in values for
* the range, [beginValue,endValue).
*
*/
@declare( "this >= 0" )
public static int sum(
@declare( "this != null" ) int[] values,
@declare( "this >= 0 && this < values.length - 1" ) int begin,
@declare( "this >= begin && this <= values.length" ) int end ) {
assert Check.sum(Enclosing.class,values,begin,end);
[/code]

for a non static it would be
[code]
assert Check.sum(this,values,begin,end);
[/code]

The post condition could be done with a wrapper
[code]
int sum=Check.invokeSum(Enclosing.class,values,begin,end);
[/code]

Other valid things you [b]can[/b] do with apt are
* generate the superclass (not useful if you already extend something other than Object, but you can coerce it. If you want to extend Widget, generate a superclass that extends Widget, and you extend the superclass.
* generate a subclass which you use (could check preconditions, invoke the method, then check the post conditions) But then you have to use the generated subclass everywhere you would have called the hand coded class.
* generate a class with static helpers which you explicitly call.

Sure, none of these are quite as clean as if the particular feature you might be implementing was "in the language".

However, that is not really the goal. The goal is to make the feature "simple and robust enough" that it will get used, for those cases where currently the overhead of hand coding the feature means it doesn't get used.

I would encourage you to have a go and explore the options for using generated code for the DBC stuff. Sure, you won't get perfection, but you can probably get something that is pragmatically worthwhile.

rreyelts
Offline
Joined: 2003-06-10
Points: 0

> some Apt processor, hiding in a jar file somewhere on your classpath could modify the ast of your code

That seems like an odd thing to be worried about. If people were concerned about what processors were being run, it would be trivial to get the tool to report that.

> assert Check.sum(Enclosing.class,values,begin,end);

1) I obviously don't like that you have to make this call.

2) If you were really going to add in preconditions, postconditions, invariants, etc... what happens when your Check class needs access to the private member variables of the calling class? Do you end up having to pass every single private member variable into the Check call? Sounds like the cure is worse than the disease to me.

> Other valid things you can do with apt are

I absolutely dislike all of those ideas. No offense. I wouldn't use dbc myself if that's what was involved. You have to ask yourself, how much additional burden are you putting on the developer? A method call is bad enough, but subclassing another class is entirely another. You have to remember that the alternative is them just writing an assert:

assert i > 5;

If the solution requires more effort than that, or has any kind of cognitive dissonance associated with it, you're going to have a hard time convincing people to adopt it.

> I can live with the static method call in LongStrings because that is heaps better than quoting multiple String
literals and concatenating them.

Not me. Between the hackiness of declaring it in the javadoc (unfortunately nothing you can do about that, and I don't know that we'll ever be able to affect the language syntax) combined with the hackiness of making a method call, I'd either use +'s or externalize the string.

God bless,
-Toby Reyelts

Message was edited by: rreyelts

brucechapman
Offline
Joined: 2004-03-18
Points: 0

> That seems like an odd thing to be worried about. If
> people were concerned about what processors were
> being run, it would be trivial to get the tool to
> report that.

Sure but once using apt becomes as ubiquitous as say reflection, would they notice the one strange viral one out of the 100 or so displayed? I think not.

> Sounds like the cure is worse than the
> disease to me.
>
> You have to ask yourself, how much
> additional burden are you putting on the developer?

> If the solution requires more effort than that, or
> has any kind of cognitive dissonance associated with
> it, you're going to have a hard time convincing
> people to adopt it.
>

Exactly. But it is still worthwhile exploring. Sometimes there will be gold, sometimes just dirt. But the gold will be hidden in the dirt until you get your hands dirty.

You cannot take one handful, analyse it and then say there is no gold anywhere here.

I was presenting some of the general ways of looking at a problem where apt might be useful.

I also am not convinced that generated code for DBC has low enough cognitive dissonance, but neither am I convinced it does not.

You can reduce the syntactic baggage with @NotNull @NotNegative @NotZero and similar for the common special cases, and maybe the general cases where you need to write a boolean expression are best coded as assert statements or some other mechanism where the expression is evaluated in situ, that way any syntax errors are identified at source, not somewhere else, and name resolution happens correctly.

As you suggest, another option is bytecode manipulation, maybe a solution is that you generate a ClassFileTransformer and manipulate the classfile at loadtime using the java.lang.instrument package. I haven't done any exploration down this track yet.

rreyelts
Offline
Joined: 2003-06-10
Points: 0

> Sure but once using apt becomes as ubiquitous as say reflection, would they notice the one strange viral one out of the 100 or so displayed?

This is running off topic, but I think that analogy is totally invalid, for all sorts of reasons. Suffice it to say that I don't think people will be using 100s of annotation processors. For the project I'm currently working on, we have 3 bytecode enhancers (JDO included), and we're approaching a million lines of code. I don't think the number of processors will explode to be 100x larger. As things stand, it's quite easy for us to keep up with what does and does not modify which bytecode. I don't think that will be changing.

> maybe the general cases where you need to write a boolean expression are best coded as assert statements

I wouldn't find that acceptable, but that's just me.

> that way any syntax errors are identified at source, not somewhere else, and name resolution happens correctly.

Yes, I now see that those are additional problems with the "generate a Check class" strategy.

>maybe a solution is that you generate a ClassFileTransformer and manipulate the classfile at loadtime using the java.lang.instrument package

Potentially, but I doubt it. It'd be much easier to either save whatever metadata you need, or to go ahead and generate the class file. In any case, this is all far removed from apt processing as we currently know it.

God bless,
-Toby Reyelts

rms7326
Offline
Joined: 2003-06-10
Points: 0

rreyelts,

Can I encourage you to use the @invariant for invariants, @require for pre-conditions, @ensure for post-conditions, and @check for others? Bertrand Meyer introduced these names (with Eifel I think) and I really like them because you can turn them off in succession as you become more comfortable with your code. In other words post-conditions could be turned off if you are confident that a library is solid. Pre-conditions could be turned off if you are confident that the caller is keeping his contracts.

With regard to your ADBC implementation, do you plan on putting it on java.net? Also, are you designing it so that the assertions are inherited automatically. This would save a lot of time and typeos that always occur when you try to propagte the contract by hand (or when it changes during development).

rreyelts
Offline
Joined: 2003-06-10
Points: 0

[i]Can I encourage you to use the @invariant for invariants, @require for pre-conditions, @ensure for post-conditions, and @check for others?[/i]

My primary focus with ADBC is providing a solution that unifies the documentation of a method's interface with its actual implementation. This is because I believe that, by far, the largest number of problems incurred in using a library are caused by:

a) passing incorrect argument values or
b) expecting wrong return values

For implementation preconditions, postconditions, and invariants, we always have "assert". Developers can use that internally and can turn it on and off at will.

I just don't believe there is enough return in subsuming the @declare annotation with @precondition and @postcondition, because it introduces more problems than it solves. For example, it will significantly increase the complexity of the syntax. As another example, it will prevent me from doing nifty things, like warn that particular parameters or return types on a method weren't annotated.

[i]With regard to your ADBC implementation, do you plan on putting it on java.net?[/i]

I typically host my own stuff locally, in conjunction with SF, but I may start putting stuff on java.net in favor of SF in the future.

[i]Also, are you designing it so that the assertions are inherited automatically.[/i]

I haven't really thought about inheritance. I'm not sure how it can work correctly, given that subclasses could provide a "widened" contract for an overriden method. It does provide some interesting possibilities - for example, you could specify the contract on an interface's methods and then they could be automatically inherited by any implementation.

God bless,
-Toby Reyelts

wangzaixiang
Offline
Joined: 2004-11-24
Points: 0

> Interesting idea. Perhaps by combining this with
> Tiger's new meta data annotations someone clever soul
> could come up with a set of "Design By Contract" tags
> such as @invariant, @require, and @ensure. Then one
> would only have to specify the contracts on an
> interface or a base class and all implementing or
> extended classes would be automatically decorated
> with the appropriate assertions. Has anyone explored
> this idea?

DBC is a powerful tool both for design and for codeing/testing, and should be include in Java. although it is able to implemented with @annotation, or even using the assert keyword, but annotation is poor on expression, define it in the language and implemented in the compiler may provide more rubust support for develop.

Personally, I think DBC is more important than Generic.

rreyelts
Offline
Joined: 2003-06-10
Points: 0

[i]someone clever soul could come up with a set of "Design By Contract" tags[/i]

How weird. I just began work on writing up my own DBC package using annotations and apt. The current name is "ADBC". Right now, it has one Annotation type, @declare, which takes a Java predicate, and can be applied to method parameters and return types. Here's an example from one of my test classes.

[code]
/**
* Returns the sum of Math.abs( value[ i ] ) in values for
* the range, [beginValue,endValue).
*
*/
@declare( "this >= 0" )
public static int sum( @declare( "this != null" ) int[] values,
@declare( "this >= 0 && this < values.length - 1" ) int begin,
@declare( "this >= begin && this <= values.length" ) int end );
[/code]

brucechapman
Offline
Joined: 2004-03-18
Points: 0

Hi Toby,

Thats cool.

1. What does the generated code look like? and how do you call the generated code?

2. I was sketching out something similar for inclusion in my rapt project here on java.net

If you don't have another "home" planned for your DBC stuff, you'd be welcome to do it under rapt, thats what it is for. Send me a message on the rapt forums or mailing list and we can discuss.

Even if you host it elsewhere, rapt may still be of interest as a place to discuss apt issues.

Bruce

rms7326
Offline
Joined: 2003-06-10
Points: 0

Perhaps there are many people interested in a "Design By Contract" project using the new meta data tags. Wouldn't it be better to concentrate the resources into a single project on java.net or apache? I haven't seen any discussions about DBC with regard to the new meta data which was a surprise especially considering how unhappy some people were with Sun's assert mechanism. It seems like the meta data facility is perfect for making a good DBC system in Java that can be completely removed from released code and doesn't require a language change.

hlovatt
Offline
Joined: 2003-11-18
Points: 0

I have written a compiler, Pattern Enforcing Compiler (PEC), that in many ways is similar to APT:

https://pec.dev.java.net/nonav/compile/index.html

It allows you to extend what javac does and under the hood it uses javac. The difference is that you are not limited to generating new class, as apt limits you to, you can also modify existing classes using the PEC.

My experience with the PEC has been very positive and this sort of idea, using annotations and alike, seems to be better than allowing new syntax to be added or allowing fundamental operators like = or . to be overloaded.

patriot1burke
Offline
Joined: 2003-06-17
Points: 0

I couldn't find this in the doco...but is it possible to run another annotation pass after javac? So, for example, you can transform the compiled .class files?

Thanks,

Bill

brucechapman
Offline
Joined: 2004-03-18
Points: 0

Not with "apt"

But you could write something and then execute the apt tool first using the programmatic interface (passing all the command line args), and once that returns, if there are no errors scan the -d destination for class files and do your stuff.

Or just write a shell/batch/cmd script to do both processes together.

vasilyp
Offline
Joined: 2004-04-22
Points: 0

It will be good to have a public compiler API and pluggability layer for compile-time processors like it done for Printing Services and ImageIO plugins (based on mainfest file or some descriptor in meta-inf directory) to force compile-time processing. So when programmer tries to compile sources using your custom framework (it should be in classpath) compiler can transparently dicover and run run framework-specific processors.

But I'm not sure about how to process sources not containing annotations (e.g. for checking class that implements custom framework interface or extends class annotated by something like mentioned design-by-contract annotations)? Is it possible with current mirror API?

brucechapman
Offline
Joined: 2004-03-18
Points: 0

> But I'm not sure about how to process sources not containing annotations

It appears from the documentation (although 1 sentence implies a contradiction), that if you use the -factory option on "apt" and the specified factory supports "*" as a supported annotation type, then a processor will be requested, with an empty atds argument.

When I find time, I will experiment to confirm this.

Bruce

jddarcy
Offline
Joined: 2004-11-02
Points: 0

Assuming that a factory which processes "*" gets queried, if there are no annotations present the factory will still be asked to provide a processor for an empty set of AnnotationTypeDeclarations. In other words, if a factory processes "*" and it gets queried, it will be asked to supply a processor regardless of whether or not annotations are present. As a corollary, even if there are no source files present, a factory that processes "*" can still get queried.

jwenting
Offline
Joined: 2003-12-02
Points: 0

And turn Java into a true interpreted language just now that we have performance on a par with the likes of C++?

Thanks but no thanks. Apt might be nice as a debugging tool but not for deploying code.

rreyelts
Offline
Joined: 2003-06-10
Points: 0

[i]And turn Java into a true interpreted language[/i]

I don't understand your comment. Is this related to my suggestion to open up writable ASTs in apt? If so, I think there is some misunderstanding. The processing that apt performs is all at development time. Apt has nothing to do with debugging, interpretation, or code deployment.

God bless,
-Toby Reyelts

gvwilson
Offline
Joined: 2004-01-26
Points: 0

There are a lot of parallels between opening up the compilation process by using 'apt', and what programmers are doing every day with open-ended, extensible frameworks like Apache. If anyone's interested, I've collected a few thoughts at http://www.third-bit.com/~gvwilson/xmlprog.html (an early version of which ran in Doctor Dobb's Journal last year).

rreyelts
Offline
Joined: 2003-06-10
Points: 0

Interesting read Greg. I have to say, I shook my head in vigorous agreement when I read your list of objections to using XML as program source code. You may see XML as the wave of the future for programming, but I certainly hope that doesn't happen.

I think the more likely evolution is that programs continue to keep the same source code representation as they do now, but that compilers start opening up their ASTs to plugin developers. The plugin could modify the AST and send it back for another round of processing by the compiler. For platforms like Java and .NET, we'll probably start to see ASTs that are language agnostic, but virtual machine specific. You gave the Xen compiler as an example of where this had already been done. I think his difficulty with opening up the AST was because he originally didn't plan on fully exposing all of it as an API. If compiler writers can plan to expose an AST as an API, I don't think it would be difficult for them to create one that would remain relatively stable.

Anyway, I agree with you in spirit, that the opening up of compilers and debuggers is going to be the future of computing. Right now, I'm having to essentially create a debugger from scratch for a complex program, because the data structures and relationships are far too difficult to deal with given the current state of the art of debuggers.

On that note, let me reiterate my plea - Sun, will you please, please, please, please, please provide plugin support for your compiler where you open up a modifiable AST.

God bless,
-Toby Reyelts

rms7326
Offline
Joined: 2003-06-10
Points: 0

Interesting idea. Perhaps by combining this with Tiger's new meta data annotations someone clever soul could come up with a set of "Design By Contract" tags such as @invariant, @require, and @ensure. Then one would only have to specify the contracts on an interface or a base class and all implementing or extended classes would be automatically decorated with the appropriate assertions. Has anyone explored this idea?

cowwoc
Offline
Joined: 2003-08-24
Points: 0

Keyword: "Design by Contract"

I am *very* for this idea!

forax
Offline
Joined: 2004-10-07
Points: 0

decompile the code of apt (com.sun.tools.apt.main.Main),
it calls javac :)

javac and apt are two access points to
the same package.

cowwoc
Offline
Joined: 2003-08-24
Points: 0

Interesting. For one thing, it would be good to have more documentation and public horrah made over APT so more people learn about it but if it really is all you've said it is then I tend to agree. Replacing javac by apt sounds like a "good thing" (tm) and you should be able to keep the command-line switches backwards compatible.

brucechapman
Offline
Joined: 2004-03-18
Points: 0

apt basically just adds a couple more command line switches, but all the javac ones work the same.

Unfortunately, to make it work half decently, you really need to add an -s switch if anything generates source, and in that case also javac's -d option really needs to be explicit.

See http://wiki.java.net/bin/view/Javatools/RaptUsingApt from my rapt project for more details.

In terms of public horrah, I have the rapt project in the tools incubator which I hope will raise apt's profile a bit. However, there is so much hooplah over tiger, I am waiting till that starts to quieten down a little before I
blow everyone's sox off with what can be done with apt.

If you want to join in the fun, I would love some help on the rapt project at https://rapt.dev.java.net/ call on over and have a squizz and use the forums or mailing list to tell us what you're good at or would like to have a go at.

Quite frankly at the moment I am spending all my time doing the infrastructure stuff for the project and not much actual code development. I have a whole pile of things to try with apt, such as closures, hiding annonymous classes (delegates?), pre-condition checking, mixins ...

I am spotting use cases in these forums (people complaining that they want something - when with apt they can probably do it themselves) faster than I can sketch out the solutions (let alone code them up). For instance, that java.net article last week about extending the language to add tasks, well that particular example can be done heaps cleaner with apt and an annotation, using existing syntax!