Skip to content

Calculate num_frames if missing from metadata #732

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

Dan-Flores
Copy link
Contributor

@Dan-Flores Dan-Flores commented Jun 18, 2025

This PR enables the instantiation of a VideoDecoder without certain metadata to resolve issue #727.

  • Without num_frames_from_content or num_frames_from_header in the metadata, the VideoDecoder will calculate the number of frames using average_fps_from_header and duration_seconds_from_header.
  • Without duration_seconds_from_header, or end_stream_seconds_from_content and begin_stream_seconds_from_content from a scan, the VideoDecoder will calculate the duration using num_frames_from_header and average_fps_from_header.

Tests in test_decoders.py

  • The return value of _get_stream_json_metadata is mocked to ensure VideoDecoder is instantiated without num_frames_from_content or num_frames_from_header.
  • Python logic evaluating the metadata in VideoDecoder instantiation and get_frames_in_range are tested.

Tests in test_metadata.py

  • test_calculate_num_frames_using_fps_and_duration instantiates VideoStreamMetadata without num_frames_from_content or num_frames_from_header, and checks that various values of average_fps and duration_seconds result in the correct number of frames.

  • In test_num_frames_fallback, the test where both num_frames_from_content and num_frames_from_header are None is no longer valid. A similar test case is added in test_calculate_num_frames_using_fps_and_duration.

  • test_duration_seconds_fallback validates that calculating metadata using a scan is prioritized over header metadata.

  • test_calculate_duration_seconds_using_fps_and_num_frames tests the fallback implemented in this PR.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Meta Open Source bot. label Jun 18, 2025
@@ -129,8 +129,15 @@ def num_frames(self) -> Optional[int]:
"""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the doc string, we should also say what we do if num_frames is missing from all of the metadata.

elif (
self.average_fps_from_header is not None
and self.duration_seconds_from_header is not None
):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just went down an interesting rabbit-hole of asking "Should we instead use the average_fps property?" And the answer is no! :) If self.average_fps_from_header is None, then self.average_fps actually calls this property, self.num_frames and we'd (I think) hit infinite recursion. I do wonder if there is a way for us to be more systematic about this, where we could more centrally define the logic of under what circumstances we use which metadata.

all_frames = decoder.get_frames_played_in_range(
decoder.metadata.begin_stream_seconds, decoder.metadata.end_stream_seconds
)
assert_frames_equal(all_frames.data, decoder[:])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we also add a simple assert len(decoder) == CORRECT_VALUE?

Copy link
Contributor

@scotts scotts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had initially approved, but it looks like the tests are failing on Mac - let's make sure we're using the right tolerances there.

@scotts
Copy link
Contributor

scotts commented Jun 20, 2025

Thank you! Also, this is only half of #727. We also want to be able to handle if num_frames is present but duration is not. This half, though, is definitely the more valuable side, as core operations in the Python VideoDecoder depend on num_frames.

@Dan-Flores Dan-Flores force-pushed the calculate_missing_num_frames branch from 8d60d07 to 3075975 Compare June 23, 2025 15:01
)
assert len(decoder) == 390

# Test get_frames_in_range
Copy link
Contributor

@scotts scotts Jun 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add a comment here that we're really just testing the logic in the Python VideoDecoder class when we do the decoding. We do checks that involve num_frames at the Python layer, and that logic will use the mocked metadata. But the C++ layer does not use that mocked metadata, and will instead use what it read from the video file itself.

We should also create an issue about adding a video file that has the number of frames missing in its metadata so we can actually test the C++ logic as well. This test would then use that video file, and not need any mocking.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point! I'll remove some unnecessary python calls here, and add a comment indicating that the python logic is the only aspect being tested.

It looks like issue #710 covers this subject.

@scotts
Copy link
Contributor

scotts commented Jun 23, 2025

Thanks, @Dan-Flores! I looked into the failures, and they're all known (#740, #734, #724).

@Dan-Flores Dan-Flores marked this pull request as draft June 23, 2025 20:16
@Dan-Flores Dan-Flores marked this pull request as ready for review June 23, 2025 21:00
@Dan-Flores
Copy link
Contributor Author

I've updated this PR to complete the other half of #727, and updated the description as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Meta Open Source bot.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants