Smooth build - build system for java projects

Shared Code
#1

Hi,

As I was fed up with maven, ant and graddle tools I implemented my own build tool / build language.
I’d like to share it with the community so it is open sourced (Apache 2 licence) available at https://github.com/mikosik/smooth-build.
Full documentation is available at http://www.smooth-build.org/.
If you are curious what it can do and why it is better than other tools you should start with short tutorial at http://www.smooth-build.org/tutorial.html.

I happily answer any questions in this thread.
If you have open source project that is available online I can write smooth build script for you for free.

Enjoy!

#2

I’ve been playing with this today, and quite like it :slight_smile: Nice and simple.

A few things I would suggest though…

  • The jar function currently names the output based on the given function. I was hoping to have an “all” where I could compile multiple files at once, but I cant do this unless I zip all the outputs into an “all” zip file. The jar function should take the name from its parent function, not the first one run, or maybe just add an optional String parameter for filename?
  • A “default” function should be checked for and run if someone just runs “smooth build”, instead of giving an error, for similar use to above.
  • At the moment, on linux atleast, it seems the results are a symlink to the final result in “.smooth/values/”, instead of moving the actual file.

Not much experience with git, maybe if I use it a lot more I’ll try contributing :slight_smile:

#3

Thanks for feedback!

[quote]The jar function currently names the output based on the given function. I was hoping to have an “all” where I could compile multiple files at once, but I cant do this unless I zip all the outputs into an “all” zip file. The jar function should take the name from its parent function, not the first one run, or maybe just add an optional String parameter for filename?
[/quote]
If you want an “all” function that builds 3 jar files you can to it this way (I skipped definition of file?.jar functions):

file1.jar: …
file2.jar: …
file3.jar: …
all: [ file1.jar, file2.jar, file3.jar ];

running “smooth build all” will create result that is a dir with all 3 files inside. This way script describes exactly your intentions: You want function (“all”) that builds 3 files, each file is built by specified function.

[quote]A “default” function should be checked for and run if someone just runs “smooth build”, instead of giving an error, for similar use to above.
[/quote]
This is something I thought about but I didn’t like the idea of “smooth build” running function with some hardcoded name. I mean “how somebody finds out that he/she should name the function ‘default’ without reading lengthy documentation?” I’m thinking about introducing some kind of annotations to the language so you could write:

@Default
all: [ file1.jar, file2.jar, file3.jar ];

developers new to that language could learn that @Default exists just by reading others code. However there’re others more important features to be implemented first (like mutlthreadig).

[quote]At the moment, on linux atleast, it seems the results are a symlink to the final result in “.smooth/values/”, instead of moving the actual file.
[/quote]
This is by design. Smooth could copy the file instead of linking to it but this would just use space on your disk and increase building time (just a little bit). Linking is save as files inside “.smooth/values” never change - name of the file is the sha1 of its content.
If this causes some problems then let me know.

#4

[quote]If you want an “all” function that builds 3 jar files you can to it this way (I skipped definition of file?.jar functions):

file1.jar: …
file2.jar: …
file3.jar: …
all: [ file1.jar, file2.jar, file3.jar ];

running “smooth build all” will create result that is a dir with all 3 files inside. This way script describes exactly your intentions: You want function (“all”) that builds 3 files, each file is built by specified function.
[/quote]
Hmm, Okay it sort of works with using the array, making a folder, although I’m not sure I like that way. I want a build all so it just builds everything, but this way it wont update the same files as running each individually, instead it makes the “all” directory with the outputs. All the outputs are also just numbers, a directory with 0, 1, 2, etc., I’m assuming its their index in the array.

A similar issue with the “build all” though, if I want to have instead a “build release.zip”, which compiles everything, creates the jars, and packages everything into the zip, it wouldn’t work, as everything inside would be named numbers, and I’m not so sure I could make directories inside. Here’s an example of what I would like to do:

all:
	game.jar & util.jar ;
release.zip:
	[ bin, file("levels"), file("config.cfg") ] | zip ;
bin:
	[ game.jar, util.jar ] | concatenateFiles(with=files("game/lib")) ;
game.jar:
	game.classes | concatenateFiles(with=files("game/res")) | jar ;
game.classes:
	files("game/src") | javac(libs=files("game/lib") ;
util.jar:
	util.classes | concatenateFiles(with=files("util/res")) | jar ;
util.classes:
	files("util/src") | javac(libs=files("util/lib")) ;

This would, ideally, make a zip file with 2 directories(“bin” and “levels”), and the default “config.cfg”. The “&” idea was just that it would create both of them in the results directory, kind of a break for the builder I guess, drop what’s before it into results, and move on.

Also, random small suggestion, is there anyway you can make the functions smarter, so they don’t need the parameter name to work? :stuck_out_tongue: Maybe have the pipe always be the first argument of its type?

#5

Your release.zip contained files which names were just numbers because output of jar,zip or similar functions is just a Blob - a raw content of the file.
You can create file if you add path to it. You can use newFile() function for that purpose (I’m not sure whether its name is quite proper).
Moreover each File is also a Blob and can be used wherever Blob is required.
So for example unzip() function requires a Blob but usually you pass a File to it (a File created by “file()” function), for example:

zip-content: unzip(file("myfile.zip"));

but you can pass Blob as well. In example below files are zipped creating a Blob which is passed to unzip function in next step (does not make sense but shows the idea).

zip_and_unzip: zip(files("src/java")) | unzip ;

Concerning your project: You can build release.zip this way:
Notice a newFile() function which is not yet documented (my mistake - I will fix it today) but is released with latest smooth so you can use it.
I didn’t have time to recreate your project on my comp so I haven’t run the code below so beware of typos.


release.zip:
   concatenateFiles( bin,  with=[file("levels"), file("config.cfg") ] | zip ;
bin:
   [ game.jar, util.jar ] | concatenateFiles(with=files("game/lib")) ;
game.jar:
   game.classes | concatenateFiles(with=files("game/res")) | jar | newFile("game.jar");
game.classes:
   files("game/src") | javac(libs=files("game/lib") ;
util.jar:
   util.classes | concatenateFiles(with=files("util/res")) | jar | newFile("util.jar");
util.classes:
   files("util/src") | javac(libs=files("util/lib")) ;

btw: concatenateFiles looks really ugly and I’m thinking about adding “+” operator for concatenating arrays.
This way you could write:

   game.classes + files("game/res") | jar | newFile("game.jar");

instead of

   game.classes | concatenateFiles(with=files("game/res")) | jar | newFile("game.jar");

Again this is something to rethink and implement which needs some time :slight_smile:

[quote]Also, random small suggestion, is there anyway you can make the functions smarter, so they don’t need the parameter name to work? Tongue Maybe have the pipe always be the first argument of its type?
[/quote]
They are smarter :slight_smile: You can skip parameters names in places where smooth can guess the proper parameters. This won’t work everywhere though.
for example:
file(path=“levels”) is equal to file(“levels”)
you can omit path name because file() function has only one parameter.
You can also omit parameters where you pass argument of some type and there’s also one parameter with that type available. It happens when function has exactly one parameter of that type or all other parameters with that type has been assigned explicitly (by specifying parameter name).

cases like this:
game.classes | concatenateFiles(with=files(“game/res”))
are a little more tough, Note that we can get rid of pipe and rewrite it this way:
concatenateFiles(game.classes, with=files(“game/res”))
which is shorter version of
concatenateFiles(files=game.classes, with=files(“game/res”))
So you can omit “files” because smooth will notice that parameter “with” is assigned explicitly so the only one left (“files”) must be assigned from the only available argument. You cannot however skip both parameter names because they have the same type so smooth won’t know which argument goes with which parameter.

There are more complicated cases but I do not want to go into details here - rule of thumb is: “Omit parameter name if you think smooth can deal without it. If it can’t it will print proper error explaining why it had problem with automatic assignments”

#6

Wait!
I’ve just realized that newFile() function has a bug (wrong type of content parameter) and won’t work as I described above.
I will fix it tomorrow and let you know in this thread.
Sorry.

#7

Getting to understand the inner workings of this a bit better from this :slight_smile:

Okay, newFile would fix most my issues, however it seems to currently only make plain text files, as the content parameter is a String. Smooth isn’t able to convert the blob to a string, so it fails. Maybe make it take a blob, and give Smooth a way to convert a string to a blob if needed?

And does the “dir” function work as I intended then, making a directory with the name dir? Or would I have to add newFile to the end of it as well?

Also, that was just something I pieced together for the post, it isn’t an actual project. Its structured based on using smooth in my workspace directory, as I use eclipse and have a “util” project for most my common code. I may change the build.smooth so that it is used inside the game’s project directory, and just add a symlink to the other project.

Thinking which, is it currently possible to “include” another build script? Would be much cleaner if I could just include the util project’s script instead of rewriting it into the game.

[quote]cases like this:
game.classes | concatenateFiles(with=files(“game/res”))
are a little more tough, Note that we can get rid of pipe and rewrite it this way:
concatenateFiles(game.classes, with=files(“game/res”))
which is shorter version of
concatenateFiles(files=game.classes, with=files(“game/res”))
So you can omit “files” because smooth will notice that parameter “with” is assigned explicitly so the only one left (“files”) must be assigned from the only available argument. You cannot however skip both parameter names because they have the same type so smooth won’t know which argument goes with which parameter.
[/quote]
I actually used the “files=” version before as well, but just preferred how it looked using pipes. The “+” method definitely seems the cleanest way to go about it though.

EDIT: Ah seems you noticed the issue with newFile, not exactly in a rush, don’t worry about it :slight_smile:

#8

[quote]Okay, newFile would fix most my issues, however it seems to currently only make plain text files, as the content parameter is a String. Smooth isn’t able to convert the blob to a string, so it fails. Maybe make it take a blob, and give Smooth a way to convert a string to a blob if needed?
[/quote]
That is probably solution I will go for. :slight_smile:

[quote]And does the “dir” function work as I intended then, making a directory with the name dir? Or would I have to add newFile to the end of it as well?
[/quote]
There’s no “dir” function and there’s no “dir” concept in smooth at all.
File is just a content + path (note that path which is just String can contain directories).
File tree hierarchy is just a map between each file-path to each file-content. Conceptually this is very similar to the way git treats files.

If you need to place a file inside some specific directory in your zip file you just use appropriate path when creating that file, for example (I assume newFile is fixed to take Blob not String as content):


blob: ....
my-zip: newFile(blob, "some/complicated/path/filename.txt") | zip ;

Also the other way around if you unzip something you will get array of files and each file will have full path (together with directories).

The only downside of it is that you cannot create empty directory. For example it is not possible to create zip that would contain empty directory - you need to create some dummy file which path contains required directory. I believe this is not a real issue.

[quote]Thinking which, is it currently possible to “include” another build script? Would be much cleaner if I could just include the util project’s script instead of rewriting it into the game.
[/quote]
When rewriting my old ant scripts I converted it from the bottom. Ant called smooth (not the other way round). This should be preferred way for migrating to smooth from other build-tools as well. Mainly because most of them have already some way of calling command-line tools. You can simply invoke “smooth build xxx” from the other build-script and expect xxx to be available in .smooth/results/xxx which can be processed further by ‘parent’ script.

#9

Sheesh I did some bad explaining, now to clarify :-X

[quote]And does the “dir” function work as I intended then, making a directory with the name dir? Or would I have to add newFile to the end of it as well?
[/quote]
When I said this, I meant “bin”. From what you’ve said though, it would seem not. In the previous code example, I wanted all the files listed in “bin” to be put into a directory named “bin”. As I can see it, the only way would be to use the following:

bin:
    [ newFile(toBlob(game.jar), "bin/game.jar"), newFile(toBlob(util.jar), "bin/util.jar") ] ;

Of course this could be done without toBlob if I removed the newFile at the end of each jar function. This is probably a much cleaner way of doing what I want.

The issue comes when wanting to move the lib files I loaded into the bin directory. At the moment I think I would have to load each seperately, until the parallel piping is implemented(I am assuming it would output results back as an array?, and even then I’m not sure how well it would work. Maybe a “moveFile(File,String)” that keeps the files name and replaces the path before it with a new string?

(EDIT: So ‘files(“game/lib”) || moveFile(“bin”)’ would change “game/lib/lwjgl.jar” to “bin/lwjgl.jar”, for instance)

[quote]Thinking which, is it currently possible to “include” another build script? Would be much cleaner if I could just include the util project’s script instead of rewriting it into the game.
[/quote]
When I said include another build script, I meant another Smooth script. I was thinking along the lines of having multiple smaller projects as part of a whole, each with their own “build.smooth”, and having each of them included into the main project’s “build.smooth” file.

#10

Sorry for late reply. Somehow I didn’t get notification about your new post in this thread.

I’ve just published smooth 0.8.0 that fixes problem with newFile() function. As you suggested I created two new functions:
Blob toBlob(String string)
File toFile(Blob content, String path)
and removed newFile() function
Full release notes as usual available on blog: http://smooth-build.blogspot.com/2013/12/smooth-build-ver-080.html

  1. changing file paths:
    You are right, smooth way of doing it would be this:
    files(“game/lib”) || moveFile(“bin”)
    I would rather name it prefixPath - feels more natural in functional world.

In current implementation you have to build it ‘manually’:
game.jar: …
util.jar: …
fileInBinDirectory: [ toFile(game.jar, “bin/game.jar”), toFile(util.jar, “bin/util.jar”) ];

[quote]At the moment I think I would have to load each seperately, until the parallel piping is implemented(I am assuming it would output results back as an array?
[/quote]
Yes, parallel piping will output array. So you could write:

png.jar: files(“data/svg”) || convertSvgToPng | zip;

  1. including sub-scripts.
    It is not currently possible but it is something I want to have. The problem is that it’s much trickier to get right than it looks. Following problems need to be solved/decided (I assume we have project/build.smooth that calls project/subproject/build.smooth).

a) when function from project/build.smooth calls function from subproject where results should be cached? in project/.smoth/values/ or project/subproject/.smooth/values/. Second option allows you to reuse cached values when you run this function directly via project/subproject/build.smooth but is more difficult to implement (Caching becomes more ‘distributed’)

b) Should included script share the same namespace (So having the same function name in both will be forbidden.) ?
Or should imported script be put in some new namespace so calls to its function would have to be prepended with the namespaces like this:
artifact-from-subproject: subproject-namespace:subproject-function();
If so where to declare the name of that namespace? Inside subproject build.smooth or in the line that includes it inside parent build.smooth?

c) Soon I will add plugin system - You will be able to write your own smooth function implemented in java and use it in your script. Plugins will use namespaces to avoid name collisions same way as scripts. Should there be one command that could include other functions no matter whether they are stored in some plugin jar file or other smooth script or should it be completely separate functionality - for example plugins could be included by placing appropriate jar files in some predefined .smooth/plugins directory.

There’s probably more problems but these three come first when I think about it. I do not want to act to hastily to implement something monstrous that would cause problems in future. Anyway there’s a few more urgent (more useful features) like parallel piping, multithreading etc. However if you convince me that script including is your favorite feature I can reorder my priorities :slight_smile:

#11

Most likely because I’m using plain quotes instead of ones with your name, this one should notify you :stuck_out_tongue:

[quote]You are right, smooth way of doing it would be this:
files(“game/lib”) || moveFile(“bin”)
I would rather name it prefixPath - feels more natural in functional world.
[/quote]
The only issue with that name to me is, taking it literally, it sounds like it would just be adding the new path to the beginning of the existing one, as opposed to removing what is there and replacing it.

[quote]The problem is that it’s much trickier to get right than it looks. Following problems need to be solved/decided (I assume we have project/build.smooth that calls project/subproject/build.smooth).
[/quote]
Ya, when thinking more about it, I realized another issue. You would have to do more than just include the scripts, as all paths within that subproject would be relative, and incorrect in the main project. Not an overly needed feature any ways.

One possible alternative, to add a “build(String root,String target)” where the root is the subproject directory, and the target is what to build. You could just run the new build process, wait for it to finish, then load it’s output as if file or files was run on it.

[quote] Anyway there’s a few more urgent (more useful features) like parallel piping, multithreading etc. However if you convince me that script including is your favorite feature I can reorder my priorities
[/quote]
No pressure from me, I agree parallel piping is definitely a more useful feature. I’m not doing anything overly elaborate myself right now, just putting in some ideas.

#12

I’m not sure whether I understood you at the very beginning.
Having two files game/lib/file1.jar, game/lib/file2.jar then the following function:
result: files(“game/lib”) || moveFile(“bin”) | zip ;
… will create zip file containing two files “bin/file1.jar” and “bin/file2.jar”.
This is because files(“some/path”) creates array of files with path that have “some/path” removed.

[quote]Ya, when thinking more about it, I realized another issue. You would have to do more than just include the scripts, as all paths within that subproject would be relative, and incorrect in the main project. Not an overly needed feature any ways.

One possible alternative, to add a “build(String root,String target)” where the root is the subproject directory, and the target is what to build. You could just run the new build process, wait for it to finish, then load it’s output as if file or files was run on it.
[/quote]
Yep, that’s another one to think about.
However “build(String root, String target)” is not needed - I can make smooth do it by default. I mean each time you invoke function from subproject then that invokation would be run in context of that subproject so all path will be evaluated against that subproject root.
If you however want to create a script with some set of functions to reuse between different projects then you have problem. This subscript won’t have access to files located in parent project. On the other hand maybe that’s just a different type of subscript that should be called “library”. So you can “include” sub-script of subproject and its functions will execute within different context that have access to sub-root only, and you can “import” a library of functions which will be executed within the context of importing script. If you wonder what’s use for a library, think about library that has implemented all functions that provides maven build system. If you have project built with maven you could simply import smooth-library with functions that can built project which file hierarchy conforms to maven convention. (Though that’s very far future.)

#13

[quote]I’m not sure whether I understood you at the very beginning.
Having two files game/lib/file1.jar, game/lib/file2.jar then the following function:
result: files(“game/lib”) || moveFile(“bin”) | zip ;
… will create zip file containing two files “bin/file1.jar” and “bin/file2.jar”.
This is because files(“some/path”) creates array of files with path that have “some/path” removed.
[/quote]
Ohhh, okay. I had assumed that it preserved the path. That makes sense then.