Should julia_home point to the root rather than the bin directory?

[twadleigh]

I'm in the process of trying to get jl_init to work without having to provide an argument for the general embedding case.

My current patch works on windows because libjulia.dll (whose path I'm using to set julia_home) lives in the bin directory there, but breaks on *nix because there it lives in a lib directory.

Of course, it's easy enough to patch the lib path to get the bin path, but it begs the question as to whether julia_home should just point to a level higher.

One fact in favor of redefining julia_home to point a level higher: the majority of references to it begin by appending "..".

[ViralBShah]

This thought has occurred to me before too, and it sounds like the reasonable thing to do.

[vtjnash]

the problem with it pointing a "level higher" is that it no longer points to anything at all. there is no particular guarantee that the file layout will have a bin/ or lib/ folder from that point. currently, JULIA_HOME is used to locate the julia binary for process spawning. without arguments, jl_init is expected to be able to find all of the pieces at their build-time relative locations (relative to the julia binary path of JULIA_HOME)

moving the sys.dll/sys.ji files to bin is also fine, although it should be unnecessary. there is never a guarantee either that it will be in the same folder as libjulia.dll, so libjulia.dll-relative paths of ../lib/julia/sys.ji and julia/sys.ji are equally valid

[tkelman]

Distributions can also customize a lot of the layout. Most places that hard-code relative paths are probably subtly wrong and should actually be using relative paths that get saved from build-time information.

[nalimilan]

@twadleigh What you want to do looks like a duplicate of #8650. We should definitely provide a simple way of getting the required information to embed Julia. @staticfloat provided a simple command to get the path to libjulia, would it work for you? julia -e "print(dirname(Sys.dlpath(dlopen(\"libjulia\"))))"

This kind of information could be made more systematically accessible.

[tkelman]

I seem to remember @waTeim was planning on working on something similar, not sure how far he got though.

[twadleigh]

This question arose from working on #9692, in which I'm attempting to get jl_init to work without having to provide an explicit julia_path argument when embedding.

As it stands, unless your embedding executable is in the same directory as the julia executable, things will be broken without an explicit argument to jl_init or a correctly specified JULIA_PATH envvar.

@nalimilan: when embedding, I can get the path to libjulia. Turns out, though, what I really need is the path to the julia exe.

At the very least, we should probably patch the embedding docs to say the julia_path argument to jl_init is effectively required.

[nalimilan]

@twadleigh OK. Looks like #9691 has the solution: use code from #5045 to determine JULIA_HOME based on the location of libjulia.so.

[tknopp]

Yes the issue @twadleigh was fighting with was the reason for #5045.
I think the most simple thing is to put sys.dll into a relative path to libjulia.dll and it should work by default for "regular" Julia distributions. IMHO jl_init should then also not have an argument in order to keep this simple.

[nalimilan]

Not only for "regular" distributions. If we use variables from Make.inc, it will work for all distributions which do not move things around after the build (which is a very bad idea).

[waTeim]

still working on it; i had to do other things for a while, but have recently returned to it. It's interesting the different perspectives coming from the windows side and the unix side, my situation is opposite of course, the important part for embedding is where is libjulia not julia the executable.

[staticfloat]

If you're embedding though, don't you already have the path to the Julia executable? How are you running the oneliner to get the path of libjulia if you don't know where the julia executable is?

[tknopp]

Why should one need to know where the julia executable is? I would say that libjulia has to be in the library search path and from then everything should simply work.

This issue might be simple to work around by hardcoding paths. But we really should have initialization as simple as

Py_Initialize();

[waTeim]

Yes in the library search path, but its not quite that easy for embedding because in many cases libjulia.so may be located in a directory that is not in the list of directories that are automatically searched (e.g. /Applications/Julia-0.3.5.app/Contents/Resources/julia/lib/julia/), the julia executable gets around this by setting rpath relative to itself and that's ok because whatever the particular layout is from one distribution to another the value to set will still be known and under control, but embeddings can be anywhere relative to the executable and the library so have to discover the lib directory and set it explicitly in the link step -Wl,rpath=what goes here?

[tknopp]

Then /Applications/Julia-0.3.5.app/Contents/Resources/julia/lib/julia/ needs to be put to the search path?

How does Python solve this?

But independent from how to link to libjulia whats wrong if sys.* would be installed relative to libjulia? Then one is all set once one has solved the linking issue. Currently one first fixes the linking issue and then traps into the jl_init issue.

[waTeim]

Assuming that sys.ji is in the same directory as libjulia, then seems like this small change -- "sys.ji" instead of NULL -- would work on OS/X and Linux. (julia_home_dir is where libjulia is located)

DLLEXPORT void jl_init(const char *julia_home_dir)
{
    jl_init_with_image(julia_home_dir, "sys.ji");
}

But on Windows, I don't know. A lot of this depends on the capabilities of the linker. Maybe on Windows, sys.ji needs be be in the same directory as julia.exe because of this?

[nalimilan]

libjulia should be moved to $libdir and versioned (libjulia.so.X.Y). External apps linking to a private library is insane. We have to assume the public status of libjulia, and bump the SONAME for every minor release until 1.0.

Then sys.ji won't be in the same directory as libjulia, but that shouldn't be an issue as long as we have the relative path. I don't think it makes a difference whether the reference is libjulia or the julia executable (as currently): what's important is that libjulia contains a relative path to find where the executable is.

[waTeim]

Careful, it's not just libjulia.so, but all of the other .so in the same directory which libjulia provides an anchor to find, and there are a lot of them. In the case of OS/X in particular, Julia can be/is distributed via Homebrew, but installing Homebrew is asking a lot for someone that just wants to try out Julia by dragging it into Applications.

You say executable, which executable do you mean? Julia the executable or the embedding app the executable. Are either of these two strictly speaking necessary?

[nalimilan]

Careful, it's not just libjulia.so, but all of the other .so in the same directory which libjulia provides an anchor to find, and there are a lot of them. In the case of OS/X in particular, Julia can be/is distributed via Homebrew, but installing Homebrew is asking a lot for someone that just wants to try out Julia by dragging it into Applications.

I don't understand the problem. I'm not proposing we stop being able to locate libjulia nor $libdir/julia at all.

You say executable, which executable do you mean? Julia the executable or the embedding app the executable. Are either of these two strictly speaking necessary?

The julia executable. But indeed if the goal is to be able to ship libjulia without the julia executable, we may consider libjulia to be the reference path instead of the julia executable.

[vtjnash]

I don't think it's a useful to ship libjulia without julia. That would
prevent making use of features like addprocs and caching of modules.

[tknopp]

I also don't see a need to ship libjulia without julia but does this mean the module caching will not be implemented in libjulia but in the executable? Would be nice to benefit from module caching in the embedding use case as well.

[twadleigh]

@staticfloat : an embedding application can use a system call to find out the path of the .so/.dll that is being called (as, e.g., in the patch in #9692).

@tknopp: an embedding application would need to know, as @vtjnash mentioned, the path to the julia exe in order for spawning to work correctly (or, more basically, for the patch in #9692, to get the test suite to pass).

[twadleigh]

Stale.