Add a julia-config script for configuration of externals #10069

waTeim · 2015-02-04T06:17:26Z

First step, related to #8757. This script is styled after various other frameworks
configuration scripts such as pg_config and python-config. Currently
there are only 2 arguments --ldflags and --cflags to allow embedding applications
to automatically have the necessary compiler flags set.

bizarro% julia /usr/local/julia/share/julia/julia-config.jl
Usage: julia-config [--cflags|--ldflags]

A workaround for current issue with jl_init as described in #9981 is provided for
by a compiler flag -DJULIA_INIT_DIR which can be used as an argument for jl_init,
however, both can work together and this approach is currently incomplete especially
w.r.t. Windows. Tested on Linux and OS/X. Uses compileropts(),

For exmaple if an embedding program embed.cpp contains the following

#include <julia.h>

int main(int argc, char *argv[])
{
    jl_init(JULIA_INIT_DIR);

    jl_value_t *res = (jl_value_t*)jl_eval_string("println(sqrt(2.0))");

    return 0;
}

it can be compiled using the following on OS/X

clang `julia /usr/local/julia/share/julia/julia-config.jl --cflags --ldflags` embed.cpp

and on Linux (note how the command has to be split as gcc fails when libraries come before source files).

gcc `julia /usr/local/julia/share/julia/julia-config.jl --cflags` embed.cpp `julia /usr/local/julia/share/julia/julia-config.jl --ldflags`

and then run

bizarro% ./a.out
1.4142135623730951

also on windows (using the msys2 environment for example)

gcc `julia /c/Julia/julia-e2ccda028b/share/julia/julia-config.jl --cflags` embed.cpp `julia /c/Julia/julia-e2ccda028b/share/julia/julia-config.jl --ldflags`

and then run (a.exe instead of a.out)

./a.exe
1.4142135623730951

Though the shortened form also works on Windows gcc because rpath is not used

gcc `julia /c/Julia/julia-e2ccda028b/share/julia/julia-config.jl --cflags --ldflags` embed.cpp

Also, running directly not as an argument to julia is possible

clang `/usr/local/julia/share/julia/julia-config.jl --cflags --ldflags` embed.cpp

tkelman · 2015-02-04T06:22:34Z

contrib/julia-config.jl

+end
+
+function libDir()
+   @windows_only dir = joinpath(Base.JULIA_HOME, "julia");


This isn't right - there's no julia subdirectory of JULIA_HOME, on Windows the executables and shared libraries are all in bin. Except that sys.ji and sys.dll are in the same lib/julia location everywhere.

Also please use 4-space indents for consistency with the rest of base.

waTeim · 2015-02-04T06:29:02Z

You're right, the windows args are generally incorrect. Will fix both. Otherwise, what about -I and -L, is that correct on windows? Is gcc used there? Or is the like /L and /I? I have a system to test with, but need a little help getting started, are the embedding docs generally correct? How about MSVC?

tkelman · 2015-02-04T06:33:30Z

I don't know whether anyone has tried embedding Julia within an MSVC application recently. If you use MinGW gcc then link flags will be -L and -l, yes. I'd say don't worry about MSVC at this point until Julia itself is better supported there. If someone is really motivated and wants to use this script they can try it along with the automake wrapper scripts that translate these flags.

waTeim · 2015-02-09T01:56:11Z

Updated, and tested on Linux and OS/X and Windows. I'm not sure what's up with AppVeyor; I couldn't check the error because the website timed out -- maybe that's what's up.

tkelman · 2015-02-09T02:50:19Z

You hit #10045 on win32, not your fault. I restarted the build. I think we should get some other people who've been embedding Julia in various different environments to have a look at this and try it out.

waTeim · 2015-02-09T03:02:30Z

Ok! How shall we let them know?

tkelman · 2015-02-09T03:08:28Z

Notifications should work. I think @armgong has been embedding Julia in R, and several people have been working on pyjulia, maybe just searching the issues list for "embedding" would give a good sense of who might find this useful.

armgong · 2015-02-09T03:24:51Z

thanks a alot, I will try it. could make this be a standalone execute file not a script?

waTeim · 2015-02-09T03:34:54Z

I think you mean run it without having to do julia julia-config.jl? Yea it can be done. I have not tested this everywhere, but it illustrates the point...

copy it

cp julia-config.jl julia-config

then edit it to define the interpreter -- apparently you don't have to explicitly qualify

$ diff julia-config.jl julia-config
0a1,2
> #!julia
>

make it executable

chmod +x julia-config

then run it (Windows example)

$ ./julia-config --cflags --ldflags
-DJULIA_INIT_DIR="C:\\Julia\\julia-e2ccda028b\\lib\\" -IC:\\Julia\\julia-e2ccda028b\\include\\julia
-LC:\\Julia\\julia-e2ccda028b\\bin -ljulia

tkelman · 2015-02-09T03:37:05Z

Right, but on Windows that'll only work inside a posix environment like Cygwin or MSYS. Nothing wrong with adding the shebang or making the file executable, but please keep the .jl extension.

tkelman · 2015-02-09T03:38:17Z

The shebang line should probably be #!/usr/bin/env julia to correctly pick up julia from wherever it might be installed, assuming it's on the path.

waTeim · 2015-02-09T03:45:22Z

works, I like it. Also tested it you use a julia not in path explicitly, it overrides the default which is nice. will update.

armgong · 2015-02-09T04:05:31Z

and how about change

function libDir()
    @unix_only return match(r"(.*)(sys.ji)",imagePath()).captures[1];
    @windows_only return Base.JULIA_HOME;
end

to

function libDir()
  return  abspath(dirname(Sys.dlpath("libjulia"));
end
)

waTeim · 2015-02-09T04:38:18Z

Hmm, sure; more correct anyway. I think abspath elsewhere rather than replace too.

waTeim · 2015-02-09T04:55:51Z

updated. I had to keep the replace, though.

tkelman · 2015-02-15T03:16:15Z

What's the status here, is this useful enough to merge? We should get more people who might want to use it to look it over.

waTeim · 2015-02-15T03:23:32Z

Well it works in all of the cases I've tried. I can start searching various issues for anything to do with this and include people. Jere are a few @tknopp @twadleigh @staticfloat @nalimilan

tkelman · 2015-02-15T04:10:13Z

Also pyjulia @jakebolewski

waTeim · 2015-02-15T14:38:21Z

Also might be useful for @randy3k and #8650

staticfloat · 2015-02-15T21:44:01Z

I don't do any embedding, but this looks good to me. Thanks for taking the time to send out this PR, Jeff.

waTeim · 2015-02-15T22:07:38Z

hey thanks! but you totally know how this is all put together even if you don't spend time doing embedding directly, and this is part of the solution #8757, which your are interested in.

tkelman · 2015-02-16T19:06:48Z

So should this be adjusting https://github.com/JuliaLang/julia/blob/master/doc/manual/embedding.rst if we want to recommend using this method?

waTeim · 2015-02-16T20:52:23Z

Hmm, well it could certainly makes things easier. It could be like "here, cut and paste this, it always works," kind of documentation, which is easy to follow. There could be created an example Makefile that uses it (here download this and mod to to suit your needs) -- I could add that.

OTOH, it's been tested by me and probably @armgong, but no one else.

Maybe something like give this new thing a try, but here's the historical documentation too?

tkelman · 2015-02-17T15:24:20Z

It could be like "here, cut and paste this, it always works," kind of documentation, which is easy to follow. There could be created an example Makefile that uses it (here download this and mod to to suit your needs) -- I could add that.

I like that idea.

Maybe something like give this new thing a try, but here's the historical documentation too?

Sure, no need to completely replace the existing documentation if it's still accurate, but now we can add this new method to it.

waTeim · 2015-02-21T21:53:39Z

Modified and re-tested on OS/X, Linux. Windows. I changed the behavior to better cooperate with makefiles, especially the the default makefile rules. This required me to change the command line approach to using xargs, but it turns out that makes it even simpler. Also updated the embedding documentation to more-or-less track the comments below.

On the command line (same for all 3 platforms)

bizarro% /usr/local/julia/share/julia/julia-config.jl --cflags --ldflags --ldlibs | xargs clang embed_example.c 
bizarro% ./a.out 
1.4142135623730951

In a Makefile (same for all 3 platforms)

JL_SHARE = $(shell julia -e 'print(joinpath(JULIA_HOME,Base.DATAROOTDIR,"julia"))')
CFLAGS   += $(shell $(JL_SHARE)/julia-config.jl --cflags)
CXXFLAGS += $(shell $(JL_SHARE)/julia-config.jl --cflags)
LDFLAGS  += $(shell $(JL_SHARE)/julia-config.jl --ldflags)
LDLIBS   += $(shell $(JL_SHARE)/julia-config.jl --ldlibs)

all: embed_example

Results in:

jeffw@dub1:~/src/julia-embed-test$ make
cc -DJULIA_INIT_DIR=\"/usr/local/julia-a1748b8263/lib\" -I/usr/local/julia-a1748b8263/include/julia  -L/usr/local/julia-a1748b8263/lib/julia  embed_example.c  -Wl,-rpath /usr/local/julia-a1748b8263/lib/julia -ljulia -o embed_example
jeffw@dub1:~/src/julia-embed-test$ ./embed_example 
1.4142135623730951

staticfloat · 2015-02-22T00:26:55Z

Very nice. Are there any other concerns pending, or is this good to merge? It looks like we've got a windows x64 failure, but that isn't due to these changes, I don't think.

waTeim · 2015-02-22T00:32:06Z

The only thing that changed from the version that passed and the one that did not is documentation. The Appveyor has something to do with linalg3.

I don't have anything more with specifically this PR right now. Maybe someone want's to check the doc changes?

jakebolewski · 2015-02-22T00:32:45Z

Nice. If this uses compileropts() it is going to need to be rebased, see JLOptions() in base/options.jl.

waTeim · 2015-02-22T00:38:41Z

rebase, my nemesis... kk checking it.

waTeim · 2015-02-22T02:20:15Z

Ok, updated, replaced compileropts() with JLOptions(), rechecked, verified and ...
WTH is with all those other commits? REBASE! Help?

staticfloat · 2015-02-22T06:52:33Z

Looks like something went wonky with a merge commit. Try running git pull --rebase origin master

under posix.

waTeim · 2015-02-22T07:15:07Z

that's essentially how this started. However, just in case, I (re)-tried that, but it failed horribly.

waTeim · 2015-02-22T07:22:13Z

took a guess and: rebase -i master, removed half of the commits (they are all duplicated) and then forced a push. Looks like it fixed things.

staticfloat · 2015-02-22T08:57:03Z

contrib/julia-config.jl

+end
+
+function ldlibs()
+    @unix_only return replace("""-Wl,-rpath $(libDir()) -ljulia""","\\","\\\\");


This should be -Wl,-rpath,$(libDir()) the comma is important.

staticfloat · 2015-02-22T09:22:03Z

After the comma is addressed, this looks good to me. Tested it out on my OSX laptop and it seemed to work just fine.

waTeim · 2015-02-22T20:14:45Z

I remember reading the ld page thinking where did the comma go and left it out because of it, but now I see it's because of the compiler that it needs to be in -- updated.

staticfloat · 2015-02-22T20:54:12Z

Yep, even more confusingly, omitting it will only break things sometimes, depending on where Julia gets installed. I'ma go ahead and merge now. Thanks for all your effort!

Add a julia-config script for configuration of externals

waTeim · 2015-02-22T22:20:42Z

Hey copy of my mis-post on #8428! Confirmed functional after merge on OS/X, Linux, Windows. Now on to #8757; I'll return to that discussion to ask questions.

tkelman reviewed Feb 4, 2015
View reviewed changes

tkelman added the embedding Embedding Julia using the C API label Feb 4, 2015

waTeim added 10 commits February 22, 2015 02:09

Added config script **julia-config** similar in concept to python-config

176aa70

Added define to set root of directory where julia searches for sysimage

7b65535

Fix behavior on windows

b00802c

Escape \ where necessary

a9396b0

Added shebang to allow julia-config.jl to be executed directly

3652762

under posix.

equate libdir with the directory that libjulia was loaded from

61e78a3

Mofified escaping and added --ldlibs to better support use in Makefiles

389a9f4

More escaping for Windows and it's backwards file separators

9d773a8

Added julia-config documentation to embedding doc.

41613f5

replace compileropts() with JLOptions()

248a8ce

waTeim force-pushed the waT/jconfig branch from 93bd449 to 248a8ce Compare February 22, 2015 07:19

staticfloat reviewed Feb 22, 2015
View reviewed changes

Added , for compiler to linker argument passing in ldflags

fa6a5f5

staticfloat added a commit that referenced this pull request Feb 22, 2015

Merge pull request #10069 from waTeim/waT/jconfig

f875a54

Add a julia-config script for configuration of externals

staticfloat merged commit f875a54 into JuliaLang:master Feb 22, 2015

waTeim mentioned this pull request Feb 22, 2015

Install and test embedding examples #8757

Closed

tkelman mentioned this pull request Feb 23, 2015

a command line option to get CFLAGS and LDFLAGS to link against libjulia #8650

Closed

Uh oh!

Add a julia-config script for configuration of externals #10069

Add a julia-config script for configuration of externals #10069

Uh oh!

Conversation

waTeim commented Feb 4, 2015

Uh oh!

tkelman Feb 4, 2015

Choose a reason for hiding this comment

Uh oh!

waTeim commented Feb 4, 2015

Uh oh!

tkelman commented Feb 4, 2015

Uh oh!

waTeim commented Feb 9, 2015

Uh oh!

tkelman commented Feb 9, 2015

Uh oh!

waTeim commented Feb 9, 2015

Uh oh!

tkelman commented Feb 9, 2015

Uh oh!

armgong commented Feb 9, 2015

Uh oh!

waTeim commented Feb 9, 2015

Uh oh!

tkelman commented Feb 9, 2015

Uh oh!

tkelman commented Feb 9, 2015

Uh oh!

waTeim commented Feb 9, 2015

Uh oh!

armgong commented Feb 9, 2015

Uh oh!

waTeim commented Feb 9, 2015

Uh oh!

waTeim commented Feb 9, 2015

Uh oh!

tkelman commented Feb 15, 2015

Uh oh!

waTeim commented Feb 15, 2015

Uh oh!

tkelman commented Feb 15, 2015

Uh oh!

waTeim commented Feb 15, 2015

Uh oh!

staticfloat commented Feb 15, 2015

Uh oh!

waTeim commented Feb 15, 2015

Uh oh!

tkelman commented Feb 16, 2015

Uh oh!

waTeim commented Feb 16, 2015

Uh oh!

tkelman commented Feb 17, 2015

Uh oh!

waTeim commented Feb 21, 2015

Uh oh!

staticfloat commented Feb 22, 2015

Uh oh!

waTeim commented Feb 22, 2015

Uh oh!

jakebolewski commented Feb 22, 2015

Uh oh!

waTeim commented Feb 22, 2015

Uh oh!

waTeim commented Feb 22, 2015

Uh oh!

staticfloat commented Feb 22, 2015

Uh oh!

waTeim commented Feb 22, 2015

Uh oh!

waTeim commented Feb 22, 2015

Uh oh!

staticfloat Feb 22, 2015

Choose a reason for hiding this comment

Uh oh!

staticfloat commented Feb 22, 2015

Uh oh!

waTeim commented Feb 22, 2015