wrapper-free interop with C/C++

[timotheecour]

Having a good C and C++ interop for Nim would be of strategic importance for wider adoption of Nim, as it would allow reusing the massive code bases out there (eg opencv, qt, SFML, ...) without having to either rewrite them or writing and maintaining wrappers.

Calypso

https://github.com/Syniurge/Calypso, a fork of ldc compiler for D, is an amazingly cool project that allows direct interface between D code and C or C++, without using any wrapper, any (because it uses clang and llvm), understands virtually all of C++ (including pre-processor, C++ templates, exceptions, etc).

A C++ class (eg opencv cv::Mat) or functions/templates can be used in D directly after import (C++) cv.Mat; without need to write or generate wrapper code for these, and templates don't need to be instantiated in order to be used: they can be used directly. We can pass/return by value, pointer, or reference, we can even derive C++ classes in D etc.

Here's a simple example importing Qt in D: https://github.com/Syniurge/Calypso/blob/master/examples/qt5/qt5demo_simple.d
Thanks to calypso, I was able to use some opencv functionality from D in non-trivial use cases involving heavy use of C++ features and it actually worked (modulo some bugs that have been fixed since then for the most part).

Nim interop

On the Nim side, we can embed C or C++ code as follows:

# wrapper
proc reopen_stdin*(): void {.noconv, importc.}
# arbitrary C or C++ code
{.emit: """#include <stdio.h>
void reopen_stdin(){ freopen("/dev/tty", "rw", stdin); }""".}

however the hard part is writing the wrapper code (especially for larger C API's or any C++ API)

For C projects, c2nim can be used but it's not based on a full C frontend (eg clang) and can quickly run into limitations, eg see CREATING A NIM WRAPPER FOR FMOD which shows a number of manual extra steps had to be employed to wrap C library FMOD.

For C++ projects, there's currently no way to automatically generate wrappers/bindings, one has to resort to tedious manual mapping of C++ classes, taking care of manual allocation/deallocation of C++ classes in Nim code, no pass-by-value causes a performance hit, etc.

is a calypso-like approach for Nim feasible in the short/medium/long term?

/cc @Syniurge @arnetheduck @Araq @dom96 I'm curious what are your thoughts on that. Nim compiles to C, C++, and some other backends (objc, js).
@arnetheduck wrote nlvm, a LLVM-based compiler for Nim which could be used as a starting point (it provides the glue layer between the AST (produced by Nim compiler) and LLVM, replacing the C output
with LLVM bitcode. It uses the llvm-c interface (source)

At the end of the day, I'd love to be able to just write:

{.cppmap, "opencv2/core/core.hpp" .} # no wrapper needed!
import(c++)  cv.Mat # import cv::Mat
import(c++)  cv._ # import everything under cv
let a=cv.Mat(rows = 100, cols = 200, type = cv.CV_8UC1) # stack allocated cv::Mat class
assert a.total() == 100*200
let b = cv.imread("image.jpg") # return by value, not by pointer

notes

• some C++ features may not map well to Nim world, but that's acceptable so long the only effect is not mapping these features in isolation
• @Syniurge what would be the simplest proof of concept? probably some simple C header file for starters, where we'd reuse clang to avoid c2nim doing the hard work of understanding C
• can we short-circuit some of the work that was needed for Calypso due to the fact that Nim can generate c++? not clear
• EDIT swig is an entirely different approach we could consider in parallel (see http://www.swig.org/Doc1.3/Extending.html); likely simpler than calypso approach but also with all the caveats of swig; see also Consider using libclang or libtooling for parsing swig/swig#918

links

• [llvm-dev] nlvm, a new LLVM-based compiler for Nim
• calypso
• nlvm
• swig
• https://github.com/AutoFFI/AutoFFI Call any C procedure from higher-level languages
• https://github.com/eliben/llvm-clang-samples/blob/master/test/test_tooling_sample.py
• EDIT https://github.com/genotrance/nimterop
• EDIT fragcolor-xyz/nimline: Wrapper-less C/C++ interop for Nim

[Araq]

c2nim can parse C++ code and has been used to wrap Urho3D, wxWidgets and Unreal Engine 4. It's true it's quite some manual work though.

[dom96]

with something like the clang parser you could likely create a pretty seamless experience. I thought about it in the past, but it would still be a lot of work (even though I've got a pretty nice clang wrapper already).

[timotheecour]

I didn't know nim2c c2nim also supported c++ (readme in https://github.com/nim-lang/c2nim only mentions C, we should fix that), however I played a bit with c2nim and threw at it some C++ syntax, there's many things on which it won't work well. Happy to file bugs if needed, however, my strong feeling is this can quickly become an infinite time sink (more precisely a 10-man year time sink, to quote Walter Bright in http://jkm.github.io/phobos/cpp_interface

Anecdotal evidence suggests that writing such is a minimum of a 10 man-year project

using clang as a library avoids having to parse/analyze C and C++ altogether (while guaranteeing 100% compatibility at least as far as generating llvm IR is concerned);
I strongly feel this would be time better spent compared to trying to make c2nim better at C++. Same exact experience as with D's version of that, htod (+ similar tools), which unsurprisingly never got to handle automatically real C++ projects.

I've got a pretty nice clang wrapper already

cool which one? related to that, in https://nimble.directory/search?query=wrapper I find:

nim-libclang
Please use libclang instead

libclang
wrapper for libclang (the C-interface of the clang LLVM frontend)

however both these lead to just nim-libclang, seems like a bug

Difficulties I see with automated conversion have more to do with lack of proper mapping in Nim for some C++ features. I feel the biggest one would be

• nested types, see [TODO] Nim should allow nested types #7449 "Nim should allow nested types"

There are other ones, but probably not as common:

• variadic templates (mapping them to macros instead of proc generics may cause issues), default generics args, name clashes due to style insensitivity

[ghost]

@timotheecour why do you call it "nim2c"? it's "c2nim" :)

[arnetheduck]

To provide C++ interop, you need to understand C++ ABI - this ABI is non-standard, full of surprises, twists and turns and many years worth of legacy features. It would be an incredible time-sink, and what you'd get in return is a bug-ridden maybe-works-if-you're-lucky implementation.

Right now, the Nim compiler remains ignorant of any such issues and delegates them to the backend and ultimately to the underlying C/C++ compiler - it has no notion of an ABI beyond the very bare minimum, exportc and a few other patches of support. Neither the Nim language nor the compiler are equipped to handle C++ interop, increasing the amount of work needed to get a feature like this done.

Steps are being taken to get closer - at least to a point where the dramatically less complicated C ABI can potentially be covered (for example with the alignment work). If there was an MVP for this feature, it would be covering at least C interop, which remains the lingua franca and lowest common denominator of interop. It is not without reason that even LLVM itself offers a C api - C++ is simply unsuitable for interop without vast surgery or resources (failed examples include "managed" C++ of Microsoft fame).

D is in a different position in that it tries to be an explicit C++ replacement and match many of its features, with a smaller impedance mismatch.

all in all, I think the imperfect brute-force approach of c2nim, warts and all, serves nim better for the time being, until nim matures as a language and gains the features to even meaningfully talk about the problem. at least c2nim is honest in its limitations.

[dom96]

Neither the Nim language nor the compiler are equipped to handle C++ interop, increasing the amount of work needed to get a feature like this done.

Perhaps I am misunderstanding you but I think this statement is completely wrong. The Nim language and compiler is a duo that offers one of the most powerful C++ FFIs out there. Nim compiles to C++ after all, why are you saying that it's not equipped to handle the interop?

[arnetheduck]

equipped to handle the interop

Nim compiles to C++ but it does not understand C++ - multiple inheritance, functors, ADL, template metaprogramming, even trivial stuff like destructors (yes, I know they're coming, and I'm guessing they'll be slightly off compared to C++) are all foreign to the Nim language. An import statement has to understand all that and generate viable alternatives. Doable, sure, but a lot of work. And that's the VS6 subset of C++ - a lot has happened since then (try wrapping a random boost lib - that's a good example of what modern idiomatic c++ code looks like).

[timotheecour]

try wrapping a random boost lib - that's a good example of what modern idiomatic c++ code looks like

• or simply c++ standard library (eg std::vector<T> or std::string)
• another example: opencv has dropped support for C a while ago which means https://github.com/dom96/nim-opencv wrapper is outdated (and very incomplete AFAIK)

/cc @dom96

one of the most powerful C++ FFIs out there

as described by @arnetheduck as well as in top post, this is very far from what you can do in Calypso, which doesn't require writing any wrapper, and understands a very large subset of C++ (enough to wrap C++ standard library, template heavy C++ opencv etc)

all in all, I think the imperfect brute-force approach of c2nim, warts and all, serves nim better for the time being

• that forces users to write (tedious, bugprone) manual wrappers in many use cases, eg wrapping C++ only libraries like opencv (C support was dropped a while back)

• c2nim uses hand-written parser (eg https://github.com/nim-lang/c2nim/blob/master/cpp.nim) ; I ran into too many issues when I tried c2nim on C++ projects

• parsing C++ is notoriously hard and a giant time-sink, which is why I'm suggesting here we explore an approach based on libclang, as done in Calypso, to completely by-pass what's already done by libclang.

now that motivation and current limitations are hopefully clear, I'd really like to focus the discussion in this issue on how to bridge the gap and make C++ interop more useful

There are 2 ways I can think of :

• auto-generated wrappers that understand C++ via adding Nim support to swig (see caveats Consider using libclang or libtooling for parsing swig/swig#918 that I also mentioned above already; this could be discussed in a separate issue)

• wrapper-free approach using libclang; that's the approach followed by https://github.com/Syniurge/Calypso and IMO the most promising one

it doesn't have to support all of C++ to be useful, ie, libclang will accept and parse correct C++ code, and cpp2nim (let's call it like that) will filter out symbols it can't handle (ie, it won't necessarily halt on 1st symbol it can't map to Nim). Over time it would translate a growing number of symbols

Note that c2nim doesn't have this property, since parsing failures can affect future translations.

[genotrance]

As the author of nimgen, I can agree that having a seamless interop with C/C++ would be great. I am a big proponent of code reuse and feel it is a no brainer to leverage established C/C++ libs.

If C/C++ -> libclang -> AST -> Nim can be done transparently by the compiler, I'd be all for it. This is ideal since the compiler keeps evolving and c2nim is lower in priority and not in sync. Some of the code it generates isn't valid Nim anymore. Further, keeping wrappers backwards compatible with multiple shipping Nim versions is challenging.

I'm also on the fence on replacing c2nim's engine with libclang. The separate tool will still be out of sync and some of the challenges are managed by running code through the preprocessor - Nimgen makes that easy. The only advantage would be to translate C/C++ code completely into Nim but I'm not a fan of that beyond wrapping since you need to inspect every generated line and it isn't scalable for large projects. It also goes against my principal of reuse - translated code won't benefit from upstream improvements and bug fixes. Manual edits will be inevitable and I doubt automating everything through Nimgen will be viable.

Meanwhile, I will also say that despite c2nim warts and the long list of open issues, I have been able to wrap quite a few C++ libs, let alone C. Check out nimgraphql for a complex example. Given nimgen is automated, the wrappers also stay up to date with minimal maintenance effort. The other benefit is that all users need to do is nimble install wrappername and everything just works so while the wrapper creation is a bit tedious, consumers have a seamless experience - at least that has been my goal with nimgen wrappers. What's sorely missing though is adding a Nim interop layer on top of wrappers.

Assuming current state of affairs, my near term wish list for the Nim compiler is as follows:

• Add full support for forward declarations - both types (across type blocks) and procs {.reorder: on.}
• Add support for enums with out of order values - C/C++ bitmasks often result in out of order values which breaks Nim and requires reordering
• Recognize popular C/C++ extended types - e.g. uint32_t, LPCSTR
• Add interop for popular C/C++ data types - e.g. std::vector

For c2nim:

• Some love for c2nim - bug fixes, keep in sync with Nim compiler
• Support enough C++ to make wrappers easy (proc signatures, types) - issues with C++ templates
• Consolidate all type declarations generated by c2nim into one chunk at the top - unless Nim compiler adds support for forward declarations outside the same type block

For nimgen:-

• Continue to add functionality to make wrapping easy and faster
• Drive fixes into c2nim, Nim, etc.
• Create and maintain more wrappers for popular libs

[krux02]

Go ahead, write a prototype.

[Araq]

Consolidate all type declarations generated by c2nim into one chunk at the top - unless Nim compiler adds support for forward declarations outside the same type block

Nim supports {.reorder: on.}.

[genotrance]

Nim supports {.reorder: on.}.

Oh good reminder - genotrance/nimgen#36. I will start using it in the future.

[timotheecour]

reorder is useful indeed for this task but needs a bit of love

[genotrance]

One more item to Nim wish list to my comment above:

• Add support for enums with out of order values - C/C++ bitmasks often result in out of order values which breaks Nim and requires manual reordering.

[timotheecour]

see also how AutoFFI iterates over all declarations parsed by libclang: https://github.com/AutoFFI/AutoFFI/blob/master/src/clang.cpp ; can't directly be used, but can be used for inspiration

[arnetheduck]

https://ziglang.org/ btw uses libclang to import C stuff. could be a source of inspiration.

parsing C++ is notoriously hard and a giant time-sink,

haha, if you think parsing C++ is hard, wait until you get to the semantics part ;)

[timotheecour]

libclang has a Sema library to help with semantic pass: https://clang.llvm.org/docs/InternalsManual.html#the-sema-library

links

• https://stackoverflow.com/questions/11510792/is-the-semantic-analysis-step-in-clang-an-essential-part-of-the-compiler
• https://stackoverflow.com/questions/32024115/how-to-get-a-list-of-all-global-declarations-of-a-c-c-program-using-clang
• http://amnoid.de/tmp/clangtut/tut.html (Tutorial 06: Doing semantic analysis with clang)

the bulk of the work would be to translate C++ semantic concepts (parsed by clang as AST with semantics attached, eg a TypeOf node) to Nim concepts; and that could be done gradually starting from the easiest (ignoring things that isnt' yet translatable), even on large complex programs

[Araq]

Well go ahead and start working on it. I have heard "c2nim is bad, I will write a better tool based on LLVM" from at least 3 people now. They never delivered...

[andrewrk]

If you use libclang, I advise using the C++ API. The C++ API is the real API, and the C API is a stable wrapper that barely scratches the surface area. It's a dead end.

The other problem we ran into is that pointers cannot be null in zig, but obviously in C they can, so we have to translate every C pointer as an optional pointer.
Also in Zig pointers have the concept of length (single item, unknown-length), and we have to translate C pointers as unknown-length. So it has become a bit awkward to deal with C pointers, and there's a proposal to add a new pointer type to correspond to C pointers.

[arnetheduck]

If you use libclang, I advise using the C++ API

.. or you can do like rust devs did and contribute to the C api such that it becomes less of a dead end. it's actually quite simple.

[dom96]

You can actually use a combination of the C and C++ APIs, I've done it and it works pretty well. The C API holds references to the C++ AST objects so you can query it for information that isn't available via the C API.

I've got a package that does this and I will open source it soon.

[genotrance]

I played around with libclang for a while but it is huge and tedious to build. I'm not sure it is the best course of action.

In my quest to find a better way, I found tree-sitter which is a language parser built by Github for Atom. It supports over 18 programming languages and parses them into a common AST format which is then being leveraged for syntax highlighting and code folding among other possibilities.

I've gone ahead and wrapped it using c2nim/nimgen and it works as expected for these 18 languages. I'm now looking into the AST format to see how it can be leveraged in our world.

If it becomes possible to convert this AST into Nim code, it will become possible to convert code from all these languages into Nim. Of course, have to be realistic - it may not be the case that there's a 1:1 mapping for every construct but it certainly seems interesting. Moreover, I'm interested in wrappers so I'm not as motivated to convert everything into Nim yet, just C/C++ headers into definitions that Nim can immediately leverage.

The question in my mind is whether this wrapper interop can be done at compile time - given tree-sitter is C code, it would have to be built into the Nim compiler to do that. Without that, it would end up being equivalent to c2nim. I cannot see a way to create a library that adds this capability via macros since the VM cannot importc at compile time.

Finally, it will be super cool to have a Nim grammar as part of tree-sitter so that existing Nim source code can be parsed and supported just as well as these other languages. I hope someone takes on that effort.

[genotrance]

I've started nimterop which builds tree-sitter into a binary and then converts the ast into Nim using macros. It's working pretty well so far and @timotheecour has also been making contributions.

I'll appreciate a review of the approach and any feedback to ensure this thing has legs. Again, my goal is only wrappers and not outright conversion of C/C++ to Nim so the scope is limited at this time.

[arnetheduck]

I'll appreciate a review of the approach and any feedback to ensure

my take would be that I prefer the wrapper gen to output a nim file for several reasons (instead of it all being hidden behind macro magic:

• text is king
  • no macro expansion tool will ever beat the simplicity of opening a text file
  • seeing the api in nim syntax is more simple than doing the same C-to-nim conversion as the tool does in your head - I don't have to context-switch to C when looking up a function)
  • can include comments (like c2nim)
  • can use search in editor to find functions
  • etc
• header files by definition rarely change - rerunning tree-sitter and friends on every compile is wasteful
• having a separate binary to distribute in order to compile your application makes it complicated - much easier to commit a generated nim file

I use c2nim currently to import the llvm-c headers - results can be seen here - it's a fairly extensive library where I'm unable to import some functions due to macro expansion not being sufficiently supported by c2nim

[dom96]

The question in my mind is whether this wrapper interop can be done at compile time - given tree-sitter is C code, it would have to be built into the Nim compiler to do that. Without that, it would end up being equivalent to c2nim. I cannot see a way to create a library that adds this capability via macros since the VM cannot importc at compile time.

Of course it can. Keep in mind that the VM can execute processes easily, this enables pretty much anything. You can write a small binary that takes as input a .c/.cpp filename and output a JSON-formatted AST, parse that in your macro and you've got wrapper-free interop.

Now that I think about it you can probably just run clang to output the AST as text and parse that. No need to use tree-sitter which I doubt comes close to clang's C++ parser.

[genotrance]

@arnetheduck - thanks for the feedback - agree with all your statements.

header files by definition rarely change - rerunning tree-sitter and friends on every compile is wasteful

Agreed - design is moving towards this. Right now, the code is run on every compile since it is still POC grade but generating a .nim file is where we will end up.

having a separate binary to distribute in order to compile your application makes it complicated - much easier to commit a generated nim file

My approach with nimgen has been to make the wrapper process seamless for a consumer. I like how I can simply nimble install any package and start using it - native Nim or wrapped. Any details are handled by the package itself - git clone, wrapping, paths, etc. However, it should not stop anyone from checking in the generated files.

@dom96 - nimterop already does a whole bunch using macros including creating the AST data structure so macros are really capable of anything! That being said, I'm working on moving most of the functionality into the binary since the VM is slower and this method involves C => tree-sitter AST => string => macro AST => Nim code which is round about. It will also work better standalone to meet @arnetheduck's use case.

Now that I think about it you can probably just run clang to output the AST as text and parse that. No need to use tree-sitter which I doubt comes close to clang's C++ parser.

I'd have loved to do this but clang isn't the default on Windows or Linux. Downloading several hundred megs to just do the AST generation will be a showstopper for most.

[arnetheduck]

My approach with nimgen has been to make the wrapper process seamless for a consumer.

yeah, good point about the "get-up-and-running-seamlessly" and the package doing all this - I just don't think we're there yet, with nimble :) ie nimble install is conceptually broken - putting what should be repository-local information in a shared/global location does not scale.

you're absolutely right that if all other stars were aligned, checking in a generated file would be deeply questionable. an additional argument to do so anyway might be that it removes the need to have the header files installed at all - a common-enough situation given that on windows you usually get just a dll, and on linux you need -devel packages - I'd argue that a more "complete" solution would download the dependency, build it etc etc, but then we have a full-blown package manager. not necessarily a bad thing, just a much larger scope - or at least not the job of the wrapper generator :)

[Araq]

IMHO this "don't commit generated files" is a huge fallacy that indirectly produces stuff like

https://gcc.gnu.org/wiki/WindowsBuilding

where you need make, perl, flex, bison installed with the right version or else you can't build it.

[Araq]

I'd have loved to do this but clang isn't the default on Windows or Linux. Downloading several hundred megs to just do the AST generation will be a showstopper for most.

That's because you are looking at this problem from the wrong angle, sorry to be blunt. Ship the generated Nim code.

[genotrance]

I'd argue that a more "complete" solution would download the dependency, build it etc etc

@arnetheduck - so that's what I have done for most nimgen wrappers - download the upstream sources via git or zip, generate wrappers and then compile in all the sources, no binary required. And none of this is the end user's problem. You simply nimble install and import X. The result is a single binary with no external dependencies.

There are some wrappers where this isn't possible (nimbass) or painful (nimssh2) but nimgen supports compiling in, DLLs and static lib scenarios.

not necessarily a bad thing, just a much larger scope - or at least not the job of the wrapper generator :)

I've taken this on in nimgen to an extent since I feel it is crucial to make the process seamless. It makes the nim ecosystem that much richer and easier to get started in. It is tedious up front but I've done 25 wrappers myself. Every package is tested with 0.18.0, 0.19.0 and devel daily with the latest upstream changes so anyone can use them at any time.

IMHO this "don't commit generated files" is a huge fallacy that indirectly produces stuff like
That's because you are looking at this problem from the wrong angle, sorry to be blunt. Ship the generated Nim code.

@Araq: I know we discussed this a bit earlier on IRC but as asked before, there's too many moving parts:

• OS - Win/Lin/Mac, others
• C/C++ compiler
• Nim version
• Upstream package improvements

I can check in a bunch of Nim files as a Nimble package but they are a snapshot in time. Anything outside that tested combination and stuff may not work or meet the consumer's requirements. You see so many Nim wrapper packages in this situation which were purpose built by the consumer for a project, not built as a sustainable package.

This approach works for a consumer - install at a point in time and save that combo of everything and maintain it in source control until there is a need to upgrade. It does not work as a library maintainer who has to cater to any variation of the above. It is not a seamless experience in just a few months and not scalable if you have to generate and maintain an archive of combinations.

Nimgen doesn't solve everything but allows me as a package maintainer to keep things up to date. If a consumer wants a snapshot, they can and absolutely should make one. These aren't reproducible builds though, not yet.

If there's a way to solve this in a scalable fashion, I'm all for it. That being said, my primary goal is to make it easy for others to use these packages and transition into Nim. I don't believe making it static is going to achieve that.

[Araq]

Here is the workflow I assumed your nimgen uses, and it still seems feasible to me. Feel free to mentally replace c2nim by a better tool.

• You have a bunch of header files with #ifdef and #define etc.
• You copy the header files and edit them wildly to make it parsable with c2nim, you use c2nim's "embed Nim code" feature (see https://nim-lang.org/docs/c2nim.html#embedding-nim-code). The goal is that c2nim produces a valid Nim file that does not have to be modified by you.
• You keep both the original header files as well as your modfiied ones around. If a new version of the header file arrives, you diff the original header file and your changed header file and re-apply the diff.
• You ship the Nim files to Nim users. If something doesn't work, you fix the bug by modifying the header file and run c2nim again.

[genotrance]

There was a long discussion on IRC about this topic. The list of concerns with the nimgen approach are the following:

• Seems fragile, can break anytime
  • Hope to continue to alleviate this with CI
  • Travis tests nimgen and all wrappers daily on Nim devel on Linux and OSX
  • Appveyor tests nimgen and all wrappers daily on 0.18.0 and 0.19.0 on Linux and Windows
• #define XYZ for crucial consts needs to be more robust
  • Nimgen right now searches for int|float|hex in source files and converts to consts - does not respect #ifdef though
  • Investigating if gcc -E -dD can help pull relevant platform specific definitions - seems to be working in limited testing
• Need reproducible builds to produce effect similar to static generated Nim files checked into source control
  • Today, user needs to install wrapper and backup for posterity which is anyway good practice
  • However, need to enable installing a specific version from the past
• #define funcn2(x, y, z) for function signatures is not possible today
  • Need to research ideas around achieving this

Thanks for all the feedback so far. Meanwhile, we are continuing to work on nimterop and will port relevant improvements into the nimgen workflow accordingly.

[genotrance]

Quick update on this issue - nimterop has been growing in functionality over time. The road is still long but the current status makes me optimistic.

I think it is time to close this issue since Nim provides all the infrastructure required to pull off this interop without compromise or requiring any fundamental changes to the compiler. Minor details have been discussed in other issues and have provided a good direction to continue on.

I encourage the community to continue providing your guidance and feedback to ensure development continues in the right direction.