Better Neuroscope support. #3862

theodchrn · 2025-04-14T12:54:18Z

Hi!
This is the first time i'm doing a pull request on an "official" project, and don't really know how to do it. If you have any remark/advice on how to do it properly i'd be glad to have them!

Basically i would just like to share my small improvements on reading neuroscope-type data (amplifier.dat coupled with amplifier.xml). I don't think this format is widely used to date, but for the few of us that still do I think this might be a step forward to using SpikeInterface more easily.

What I've added for my personal use and think might benefit everyone:

fetches the xml with channel groups
automatically adds it as a group in the recording
parses the real colors and positions (offsets) directly from Neuroscope and adds them as properties of the object (handy for ephyviewer)

Will allow for parallel spike sorting, processing... as if it were separate recordings/structures (for now it's only anatomy-based, but we could do the same with the "spike groups" identified after spike sorting by Neuroscope).

- fetches the xml with channel groups - automatically adds it as a group in the recording - parses the real colors and positions (offsets) directly from Neuroscope and adds them as properties of the object (handy for ephyviewer) Will allow for parallel spike sorting, processing... as if it were separate recordings/structures (for now it's only anatomy-based, but we could do the same with the "spike groups" identified after spike sorting by Neuroscope).

for more information, see https://pre-commit.ci

zm711 · 2025-04-14T13:36:16Z

Thanks for the contribution @theodchrn. Super cool. I think @h-mayorquin recently did something similar for our Intan reading capabilities. Maybe he would be good to give this a glance over?

h-mayorquin

Thanks for your contribution.

I am unfamiliar with Neuroscope so I just made some general comments.

Questions:
what do the channel groups mean in this format? what do they stand for? Same for discarded channel.

For the maintainers.
Maybe both here and in Intan we should have a light version and only load the groups -and everything else- when all_annotations is true. This last suggestion should not stop your PR and we can do this after, this message is intended for other maintainers.

h-mayorquin · 2025-04-15T16:05:11Z

src/spikeinterface/extractors/neoextractors/neuroscope.py

+
+        return channel_groups, kept_channels, discarded_channels, anatomycolors
+
+    def split_recording_by_channel_groups(self):


should be private and have a name other than split which has a different meaning on SI. Maybe set_groups

what do you mean by private ? i'm not super familiar with advanced objects. Is it the __split_recording caracter ?

private in general would be
_set_groups so one underscore. The __ leads to name mangling, which I personally hate. But _ underscore is a flag to the user that this function is for internal use and that the implementation and arguments could change at any time.

ok sounds fair! i'll push a new commit for that

src/spikeinterface/extractors/neoextractors/neuroscope.py

h-mayorquin · 2025-04-15T16:12:56Z

src/spikeinterface/extractors/neoextractors/neuroscope.py

+        discarded_ppty = np.full(n, False, dtype=bool)
+        discarded_ppty[discarded_channels] = True
+        self.set_property("discarded_channels", discarded_ppty)
+        self.set_property("colors", values=list(colors.values()), ids=list(colors.keys()))


Unsure about adding colors here but I think that @samuelgarcia and @alejoe91 are more familiar with viewers so I defer to them.

It's mostly a gimmick for reproducibility of data visualisation with my lab ahah. I use it later in a custom ephyviewer script (not shown here, but maybe i should share it as well ?)

I think we just need to understand is this something all users should think about or is this super specific. If colors are always generated in a consistent (at least to the user) way then it might be worth adding (again with Heberto's final suggestion to fit this in with all_annotations for a future PR).

Basically we want properties set here to be truly meaningful to all users. Other properties can be set on a per-user basis with our property setting machinery.

Yes! In neuroscope it's always generated in a consistent manner, hardcoded in the paired xml file.

OK for color but maybe we could rename this to neuroscope_color no ?

theodchrn · 2025-04-16T11:14:56Z

Hey @h-mayorquin ! Thanks for the review :) I'll try my best to answer those.

Channel groups are basically user-derived channels segmentation, e.g. channel ids for the hippocampus, PFC, ACC... They each have their own "group" and group-color in neuroscope.
Discarded just means that the user chose not to visualize the channels in the final recording - eg ref, ground, noisy signal...
about the maintainers: the update i'm suggesting only comes because I am using "old" data that has already been processed and visualized through neuroscope - otherwise I would simply start from scratch. So this all_annotations = True would be super handy for people like me!

h-mayorquin · 2025-04-16T14:46:26Z

Thanks for your answers.

Channel groups are basically user-derived channels segmentation, e.g. channel ids for the hippocampus, PFC, ACC... They each have their own "group" and group-color in neuroscope.

Can they be anything else than brain areas? if not, then the property should be named "brain_area". "groups" in SpikeInterface are usually vaguely things that you would sort separately (e.g. a shank in a multi shank neuropixel probe).

Discarded just means that the user chose not to visualize the channels in the final recording - eg ref, ground, noisy signal...

Mmm I am unsure about this. Is this general neuroscope or specific to your lab workflow?

theodchrn · 2025-04-16T15:02:52Z

Thanks for your answers.

Channel groups are basically user-derived channels segmentation, e.g. channel ids for the hippocampus, PFC, ACC... They each have their own "group" and group-color in neuroscope.

Can they be anything else than brain areas? if not, then the property should be named "brain_area". "groups" in SpikeInterface are usually vaguely things that you would sort separately (e.g. a shank in a multi shank neuropixel probe).

Yes it could be something else than brain area - for example in my lab we have EKG and EMG recordings as well, you could probably add breathing data... I called them groups because (at least in my use case) i run the sorting in SpikeInterface based on those groups (they could represent a shank!) - I just skip the groups that are not brain areas.

Discarded just means that the user chose not to visualize the channels in the final recording - eg ref, ground, noisy signal...

Mmm I am unsure about this. Is this general neuroscope or specific to your lab workflow?

Tbh i dont know. I assumed it was general neuroscope, but maybe it's only us! In my opinion it's also one of the main advantages of Neuroscope, as you directly visualize relevant data, already sorted for the different brain/body areas you're recording.

Let me know if this is not clear enough! English is not my 1st language and I am not very familiar with neuroscope usage outside of of my lab scope.

h-mayorquin · 2025-04-16T16:14:33Z

Yes it could be something else than brain area - for example in my lab we have EKG and EMG recordings as well, you could probably add breathing data... I called them groups because (at least in my use case) i run the sorting in SpikeInterface based on those groups (they could represent a shank!) - I just skip the groups that are not brain areas.

Then, maybe we should call them neuroscope_groups (or something like that) just to distinguish it from groups.

It would be nice if we could have more meaningful names but it seems that those properties mean different things for different users.

I need to think more about this but i would like someone who works with visualization or knows neuroscope to review this as I think we need more knowledge for this system so we don't overfit to a visualization workflow or add things that are not meaningful in other contexts.

Maybe a compromise so the user can have facilities but at the same time we don't make strong commitments is to add a new method "prepare_neuroscope_for_ephyviewer" which loads these properties but the user needs to call explicitly (I am not sure this work with ephyviewer). I am slightly against adding yet another keyword because it will crash with all_annotations

samuelgarcia · 2025-06-02T08:10:59Z

src/spikeinterface/extractors/neoextractors/neuroscope.py

+        channel_groups, kept_channels, discarded_channels, colors = self._parse_xml_file(self.xml_file_path)
+        for group_id, numbers in enumerate(channel_groups):
+            group_ids[numbers] = group_id  # Assign group_id to the positions in `numbers`
+        self.set_property("group", group_ids)


I agree with @h-mayorquin for neuroscope_group

samuelgarcia · 2025-06-02T08:13:57Z

Hi @theodchrn
Thanks a lot for this.
I am OK with this.

- method now need to be called by the user

theodchrn · 2025-06-04T15:13:26Z

Hi
Apologies for the very late replies, I completely forgot to check my github.
You should find the appropriate changes in the last commit.

h-mayorquin

Now that this is divided into a function that can needs to be called intentionally I am OK with it.

I still think that it would be better to be specific about the group being neuroscope group but I am not sure if this will still work with the viewer.

src/spikeinterface/extractors/neoextractors/neuroscope.py

theodchrn · 2025-06-16T11:23:17Z

Hi, just so you know the checks are failing for windows, when building wheel for pyedflib. I don't fully understand if it's linked to my changes or not - I don't see why it would be.

zm711 · 2025-06-16T11:26:33Z

I don't think it is. We will have to dig into what changed about the package on windows. We recently had to fiddle with versioning for that library so maybe they changed something again and we will need to adjust again. But need to figure out if this is our level or a neo level issue.

theodchrn · 2025-06-16T11:31:11Z

Ok thanks for the confirmation:)

zm711 · 2025-06-16T17:22:06Z

Hopefully we have the fix in main now. Just updated your branch so we will see :)

zm711

Also good by me now if @samuelgarcia or @alejoe91 want to do a final review.

theodchrn and others added 6 commits April 14, 2025 14:48

[pre-commit.ci] auto fixes from pre-commit.com hooks

588c550

for more information, see https://pre-commit.ci

fix bad copy-paste --> now the arguments to split by group are good.

36953de

[pre-commit.ci] auto fixes from pre-commit.com hooks

b969a9d

for more information, see https://pre-commit.ci

forgot to add xml_file_path as Neuroscope attribute

fa30844

[pre-commit.ci] auto fixes from pre-commit.com hooks

d8475eb

for more information, see https://pre-commit.ci

zm711 added the extractors Related to extractors module label Apr 14, 2025

h-mayorquin reviewed Apr 15, 2025

View reviewed changes

resolve comment about np setdiff1d

9b0595f

rename split recording groups with private name

963a811

samuelgarcia reviewed Jun 2, 2025

View reviewed changes

theodchrn added 2 commits June 4, 2025 16:59

Merge branch 'main' into SpikeBase

2ba186c

fix: - change the channel and color property name to neuroscope_groups

1313c55

- method now need to be called by the user

h-mayorquin approved these changes Jun 4, 2025

View reviewed changes

src/spikeinterface/extractors/neoextractors/neuroscope.py Outdated Show resolved Hide resolved

alejoe91 added this to the 0.103.0 milestone Jun 11, 2025

theodchrn added 2 commits June 13, 2025 15:27

Merge branch 'main' into SpikeBase

ba2688b

change group attribute to neuroscope group for more clarity

63af579

Merge branch 'main' into SpikeBase

cf1c332

Merge branch 'main' into SpikeBase

7630b33

zm711 approved these changes Jun 19, 2025

View reviewed changes

Merge branch 'main' into SpikeBase

12961a6


		return channel_groups, kept_channels, discarded_channels, anatomycolors

		def split_recording_by_channel_groups(self):

Better Neuroscope support. #3862

Are you sure you want to change the base?

Better Neuroscope support. #3862

Conversation

theodchrn commented Apr 14, 2025

Uh oh!

zm711 commented Apr 14, 2025

Uh oh!

h-mayorquin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

theodchrn commented Apr 16, 2025

Uh oh!

h-mayorquin commented Apr 16, 2025

Uh oh!

theodchrn commented Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

h-mayorquin commented Apr 16, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

samuelgarcia commented Jun 2, 2025

Uh oh!

theodchrn commented Jun 4, 2025

Uh oh!

h-mayorquin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

theodchrn commented Jun 16, 2025

Uh oh!

zm711 commented Jun 16, 2025

Uh oh!

theodchrn commented Jun 16, 2025

Uh oh!

zm711 commented Jun 16, 2025

Uh oh!

zm711 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

theodchrn commented Apr 16, 2025 •

edited

Loading