[YouTube] Workaround Shorts UI for playlists by using a continuation for initial items #1104
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AudricV
added
enhancement
AudricV
force-pushed
the
yt_continuations-playlists-shorts-ui
branch
from
bccde47 to
2e3f6b4
Compare
1 task
AudricV
force-pushed
the
yt_continuations-playlists-shorts-ui
branch
from
2e3f6b4 to
b7ea744
Compare
AudricV
force-pushed
the
yt_continuations-playlists-shorts-ui
branch
from
b7ea744 to
e797b4b
Compare
query parameters This can be used to pass Google APIs standard parameters to requests.
YouTube doesn't return currently a continuation, if applicable, for Shorts UI playlists, restricting access to the 100th first items. The reel items returned don't give also upload date, uploader info and precise view count. Using a continuation which requests the first page of the playlist allows currently to get access to continuations, if applicable, and also standard video elements instead of Shorts ones, making extraction of upload date, uploader info and precise view count again possible. This method is used for all playlist types, the original request is still made, but now only returns what we need. It requires to add a protocol buffer definition file, for which its structure is based on reverse engineering of playlists continuations sent by WEB InnerTube client, received from InnerTube responses. Java classes of this file are generated for the Java Lite runtime of Protobuf with the Protobuf Gradle plugin, as the lite version is enough for our use cases. This plugin ships in JARs Protobuf definitions, which should be avoided. As Protobuf classes are parsed by Checkstyle checks and do not follow our style rules at all, an exclusion rule has been for them.
…dItems As continuations are now again available for playlists with Shorts UI, more items' tests can be enabled.
AudricV
force-pushed
the
yt_continuations-playlists-shorts-ui
branch
from
e797b4b to
43082e1
Compare
Stypox
reviewed
Apr 18, 2025
There was a problem hiding this comment.
Have you tried this to exclude the *.proto from the .jar? google/protobuf-gradle-plugin#390
Other than that, code looks good to me. Thanks!
.../java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubePlaylistExtractor.java
Show resolved
Hide resolved
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR replaces fetching playlist items with an initial playlist continuation, in order to workaround the Shorts UI in playlists, restricting access to only the 100th first items, as no continuation is currently provided. Reel items returned don't also give upload date, uploader info and precise view count.
This required to add a protocol buffer definition file, for which its structure is based on reverse engineering of playlists continuations sent by WEB InnerTube client, received from InnerTube responses.
Java classes of this file are generated for the Java Lite runtime of Protobuf with the Protobuf Gradle plugin, as the lite version is enough for our use cases. This plugin ships in JARs Protobuf definitions, which should be avoided (I didn't found something for NewPipeExtractor working on assemble Gradle tasks, only for NewPipe).
As Protobuf generated classes are parsed by Checkstyle and do not follow our style rules at all, an exclusion rule has been for them.
As a result of this change,
testMoreRelatedItemsmethod ofYoutubePlaylistExtractorTest.ShortsUItest class has been enabled.Fixes #1273.