Skip to content

Stream camera frame to ObjectDetector? #173

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
lucksp opened this issue Nov 21, 2024 · 4 comments
Open

Stream camera frame to ObjectDetector? #173

lucksp opened this issue Nov 21, 2024 · 4 comments
Labels
enhancement New feature or request

Comments

@lucksp
Copy link

lucksp commented Nov 21, 2024

I have been using React-Native-Vision-Camera + react-native-fast-tflite. Unfortunately, I need to investigate an alternate solution to passing images to the model. This happens seamlessly with RNFastTFLite. I noticed with this MLKIt library, that a uri path "string" is needed to pass to the model. That would require converting the frame from RNVisionCamera's frameProcessor to an image on the device and then create a temp file path. This is going to be very memory & storage intensive, so not a good idea.

Does this MLKIt work well with frame streams, or is it designed to be more for single image processing? I see you can set detectorMode: 'stream' option for when initializing the original model options, but it still requires a string path?

@Rvey
Copy link

Rvey commented Dec 7, 2024

Yeah same issue

@trevor-coleman
Copy link
Collaborator

This is a good question -- I think there is likely a way to implement this, but it'll take some work.

MLKit definitely supports streaming, but we'd need to implement a module interface for streams. We should probably remove the option in the meantime, to prevent confusion.

@trevor-coleman
Copy link
Collaborator

trevor-coleman commented Feb 19, 2025

Took a look and this could be implemented by writing a Frame Processor for something like react-native-vision camera.

I'm going to take a swing at writing one that we can export from the module. But they are not all that difficult to implement.

@lucksp
Copy link
Author

lucksp commented Feb 19, 2025

Took a look and this could be implemented by writing a Frame Processor for something like react-native-vision camera.

I'm going to take a swing at writing one that we can export from the module. But they are not all that difficult to implement.

Cool! If there’s anything I can do to help let me know. Would be a learning experience for me too.

Yes a frame processor makes sense!
Now I just need to figure out why my TFLite model isn’t working on your latest release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants