feat: add custom attributes to fn items

[oberrich]

This PR adds functionality to add custom attributes to function items (proposed in #2978).
I've implemented this by extending the TypeKind used by AttributeInfo to include the Function(FunctionKind) variant (see AttributeItemKind).

Unifying the callback this way is a breaking change when --with_attribute_custom or matching on the kind field is used.

[oberrich]

I'd love to add some tests as well but I'm not sure how I would do that, can't get anything other than basic tests to run.
Some of the tests seem to be Linux only and depend on some bash script.
Would love some advice although I don't think we necessarily need tests for this since I've basically just copied the logic from @mkroening's implementation and this should just work™.

[mkroening]

You can take a look at #2866, where I added some tests along with the support for custom attributes on other items (attribute-custom.h, attribute-custom-cli.h, and their respective expected output).

[oberrich]

I will be adding further changes to this PR.

The add_attributes callback isn't sufficient to generate bindings with #[cfg(...)] or #[link(...)] attributes due to type aliasing.
It also results in a lot of duplicate as well as unnecessary attributes.

I've made the following changes already:

• Codegen will track attributes for each item inside CodegenResult in a HashMap<ItemId, HashSet<String>>
• Codegen will apply attributes relevant to compilation, currently only #[cfg(...)] and #[link] are handled
• Changed callback to include information about current attributes

-    fn add_attributes(&self, _info: &AttributeInfo<'_>) -> Vec<String> { vec![] }
+    fn process_attributes(&self, _info: &AttributeInfo<'_>, _attributes: &mut HashSet<String>) {}

Example usage:

fn process_attributes(&self, info: &bindgen::callbacks::AttributeInfo<'_>, attrs: &mut HashSet<String>) {
   match info.kind {
      AttrKind::Var => {
         attrs.insert(r#"#[cfg(feature="statics")]"#.to_owned());
      }
      AttrKind::Struct | AttrKind::Union | AttrKind::Enum => {
         attrs.insert(r#"#[cfg(feature="structs")]"#.to_owned());
      }
      AttrKind::Function(_) => {
         attrs.insert(r#"#[cfg(feature="functions")]"#.to_owned());
         attrs.insert(if info.name.starts_with("NtUser") {
            r#"#[link(name = "ntuser")]"#
         } else {
            r#"#[link(name = "ntdll")]"#
         }.to_owned());
      }
   }
}

Example output with merge_cfg_attributes:

#[cfg(feature = "functions")]
pub mod unsafe_extern_C_cfg_feature_functions_link_name_ntdll {
   #[link(name = "ntdll")]
   unsafe extern "C" {
      pub fn NtCallbackReturn(
         OutputBuffer: PVOID,
         OutputLength: ULONG,
         Status: NTSTATUS,
      ) -> NTSTATUS;
   // ...
}
#[cfg(feature = "functions")]
pub use unsafe_extern_C_cfg_feature_functions_link_name_ntdll::*;

• Added a new merge_cfg_attributes post-processing pass that combines items with a unique combination of attributes relevant to compilation into a #[cfg(...)] pub mod { items... } and re-exports them via #[cfg(...)] pub use mod::*;

I still have to restore CLI functionality, we can possibly even add replace-attribute-* options and implement them using the new callback.

Progress can be tracked at https://github.com/oberrich/rust-bindgen/tree/alias for now, I will be merging them into this branch once it's ready for a review.

[oberrich]

I've rebased my changes onto upstream to simplify merging and cleaned up the commit history to make the changes more digestible.

@jschwe, I noticed you've merged #3103, and I'm wondering about the usefulness of process_comment in its current form. Shouldn't we invoke all callbacks so they can be layered?

We could take a similar approach to process_attributes and introduce a process_comments function that takes &mut Vec<String>. This would also help address #3073.

The process_comments callbacks would be invoked before process_attributes, acting as a subset of the process_attributes callback. Alternatively, we could consider removing the callback entirely, since its functionality is now largely covered by process_attributes and starts_with("#[doc"), though that might negatively impact UX.

I'm open to feedback on the best approach—would love to hear any thoughts!

[jschwe]

@jschwe, I noticed you've merged #3103, and I'm wondering about the usefulness of process_comment in its current form. Shouldn't we invoke all callbacks so they can be layered?

Just to clarify: that PR only changed the behavior from using the last registered callback (where the implementation might very well be the default None), to the last callback that actually returns something.

I find it very convenient to setup a default global implementation to process comments, e.g. converting doxygen to markdown, and then for specific header files specify another callback, which overrides the global implementation and perhaps works around some issues for that specific header file.

I also don't really see a use case for layering different comment processing steps via different callbacks. Is there something that can't be done in one callback, or would be more convenient? I think it would be much harder to reason about, since the order of the steps matter.

This would also help address #3073.

I think that issue is easily fixable without any changes. Let me check.
Edit: Opened #3114

The process_comments callbacks would be invoked before process_attributes, acting as a subset of the process_attributes callback.

For my use-case at least I would prefer getting the original comment, since parsing doxygen comments is easier, then the post-processed markdown I generate in my case.

[oberrich]

@emilio Since you're merging PRs kindly pinging again for #3088 (comment)
If this patch looks good apart from the remaining issue (incorrectly emitting an attribute here and there) and 1-2 test cases, most of the changes are implemented.
I'll then fix the tests and make sure the remaining issues are fixed and add some documentation so this can get merged.
Will most definitely squash commit history and rebase everything; beware this is a larger patch and contains some refactoring of codegen.

Looking forward to a preliminary review.

[oberrich]

@emilio @pvdrz I've now been waiting on a review for over 2 months. I understand being busy and all but not getting a response from the Rust Foundation for over 2 months is frustrating and gives me little hope for this PR ever getting merged.

Users are eagerly waiting for these changes and I'm blocked due to mismanagement...

[oberrich]

@rust-lang / @rust-lang-owner maybe more resources should be allocated to this very key piece of software? This is a necessary and long-awaited feature, yet I'm being ghosted by the organization since last Christmas [sic].

I don't think this is how contributors should be treated and hope this gets addressed quickly, multiple projects are blocked on this change. Despite having little time of my own, I made time to work on this change; as of now, I see now way forward for this change other than action on your side.

Please let me know, if this is good to be merged once the remaining issues are resolved. I do understand that this is a rather large change and can take SOME time to review, certainly not 5 months of radio silence, however.

The situation is very frustrating, I'm not sure why @emilio or @pvdrz refuse to even comment on this PR; I'm trying to stay positive here and not infer any malice, for now.

[pvdrz]

👋 I'd like to clarify some points here to (hopefully) avoid any confusion.

So first of all, I understand your frustration as I've been several times in your position as well and it is not nice. So, I think I speak for the both of us (Emilio and I) when I say I'm sorry.

However, this is a project where all its maintainers dedicate their free (both as in non-busy and non-paid) time to it. We just maintain this project out of personal interest and maybe some guilt to see it go unmaintained for extended periods of time. Sadly, that means that whenever something else happens in our lives, the project goes unmaintained because simply there's no one to take care of the influx of PRs, issues and whatnot.

Personally, I haven't found myself with time and the right head space to review any code on this project since the end of last year for personal reasons that I don't have the obligation to disclose here so I won't. I suspect Emilio finds himself in a similar position. So, you're right to not infer any malice here. We're simply not available.

On a more technical front. I also think this is an important feature as it could fix several issues that otherwise would require a custom fix instead of having a general mechanism. However, I feel hesitant to review a PR with almost 70 commits, more than 1000 line changes, merge conflicts and without a passing CI check. I'm unsure if this came to be as a lack of acknowledgement from us or not.

[oberrich]

On a more technical front. I also think this is an important feature as it could fix several issues that otherwise would require a custom fix instead of having a general mechanism. However, I feel hesitant to review a PR with almost 70 commits, more than 1000 line changes, merge conflicts and without a passing CI check. I'm unsure if this came to be as a lack of acknowledgement from us or not.

I've re-applied all changes on clean master tree, when I requested a review earlier.

Personally, I haven't found myself with time and the right head space to review any code on this project since the end of last year for personal reasons that I don't have the obligation to disclose here so I won't. I suspect Emilio finds himself in a similar position. So, you're right to not infer any malice here. We're simply not available.

I can understand this very well, I've recently been diagnosed with bipolar disorder.

Maybe @jschwe and someone else wants to band together to create a fork where we can settle details more easily and send in a more complete product for a single review. I understand the changes here are massive as they touch codegen in a lot of places and attributes play into every aspect of code.

More than happy to help as much as I can, my life is also busy, but it helps me to work a lot; I'm a bit of a workaholic, although I work on a massive amount of projects at a single time, so progress might be in weeks/months territory, before we can send in a meaningful version.

If you want to help contribute, feel free to reach out at [email protected]; we can discuss details there and establish direct comms off-the-hub.

P.S.: Anyone is welcome to contribute, the examples I mentioned are only because they've been contributing a lot already.