Skip to content

Commit c0838af

Browse files
CodeHeelerCodeHeeler
committed
Add one-shot upload
Adds one-shot upload feature, optionally specifying a repo Updates tests and docs accordingly fixes #4396 https://pulp.plan.io/issues/4396
1 parent 49b1249 commit c0838af

File tree

12 files changed

+386
-130
lines changed

12 files changed

+386
-130
lines changed

‎CHANGES/4396.feature

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
Users can upload a file to create content and optionally add to a repo in one step known as
2+
one-shot upload

‎docs/_scripts/base.sh

100644100755
+31-10
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,34 @@
1-
export BASE_ADDR=http://localhost:24817
2-
export CONTENT_ADDR=http://localhost:24816
1+
#!/usr/bin/env bash
32

4-
wait_for_pulp() {
5-
unset CREATED_RESOURCE
6-
local task_url=$1
7-
while [ -z "$CREATED_RESOURCE" ]
3+
echo "Setting environment variables for default hostname/port for the API and the Content app"
4+
export BASE_ADDR=${BASE_ADDR:-http://localhost:24817}
5+
export CONTENT_ADDR=${CONTENT_ADDR:-http://localhost:24816}
86

9-
do
10-
sleep 1
11-
export CREATED_RESOURCE=$(http $BASE_ADDR$task_url | jq -r '.created_resources | first')
12-
done
7+
# Necessary for `django-admin`
8+
export DJANGO_SETTINGS_MODULE=pulpcore.app.settings
9+
10+
# Poll a Pulp task until it is finished.
11+
wait_until_task_finished() {
12+
echo "Polling the task until it has reached a final state."
13+
local task_url=$1
14+
while true
15+
do
16+
local response=$(http $task_url)
17+
local state=$(jq -r .state <<< ${response})
18+
jq . <<< "${response}"
19+
case ${state} in
20+
failed|canceled)
21+
echo "Task in final state: ${state}"
22+
exit 1
23+
;;
24+
completed)
25+
echo "$task_url complete."
26+
break
27+
;;
28+
*)
29+
echo "Still waiting..."
30+
sleep 1
31+
;;
32+
esac
33+
done
1334
}

‎docs/_scripts/upload.sh

+17
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
pclean
2+
prestart
3+
4+
source ./base.sh
5+
6+
# Upload your file, optionally specifying a repository
7+
export TASK_URL=$(http --form POST $BASE_ADDR/pulp/api/v3/python/upload/ file@../../shelf_reader-0.1-py2-none-any.whl filename=shelf_reader-0.1-py2-none-any.whl | \
8+
jq -r '.task')
9+
10+
wait_until_task_finished $BASE_ADDR$TASK_URL
11+
12+
# If you want to copy/paste your way through the guide,
13+
# create an environment variable for the repository URI.
14+
export CONTENT_HREF=$(http $BASE_ADDR$TASK_URL | jq -r '.created_resources | first')
15+
16+
# Let's inspect our newly created content.
17+
http $BASE_ADDR$CONTENT_HREF

‎docs/_scripts/upload_with_repo.sh

+20
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
pclean
2+
prestart
3+
4+
source base.sh
5+
6+
source repo.sh
7+
8+
#Upload your file, optionally specifying a repository
9+
export TASK_URL=$(http --form POST $BASE_ADDR/pulp/api/v3/python/upload/ file@../../shelf_reader-0.1-py2-none-any.whl filename=shelf_reader-0.1-py2-none-any.whl repository=$REPO_HREF | \
10+
jq -r '.task')
11+
12+
wait_until_task_finished $BASE_ADDR$TASK_URL
13+
14+
# If you want to copy/paste your way through the guide,
15+
# create an environment variable for the repository URI.
16+
export CONTENT_HREF=$(http $BASE_ADDR$TASK_URL | \
17+
jq -r '.created_resources | first')
18+
19+
#Let's inspect our newly created content.
20+
http $BASE_ADDR$CONTENT_HREF

‎docs/workflows/upload.rst

+64-29
Original file line numberDiff line numberDiff line change
@@ -1,49 +1,84 @@
11
Upload Content
22
==============
33

4-
Upload a file to Pulp
5-
---------------------
4+
One-shot upload a file to Pulp
5+
------------------------------
66

7-
Each artifact in Pulp represents a file. They can be created during sync or created manually by uploading a file::
7+
Each artifact in Pulp represents a file. They can be created during sync or created manually by uploading a file via
8+
one-shot upload. One-shot upload takes a file you specify, creates an artifact, and creates content from that artifact.
9+
The python plugin will inspect the file and populate its metadata.
810

9-
$ export ARTIFACT_HREF=$(http --form POST $BASE_ADDR/pulp/api/v3/artifacts/ file@./shelf_reader-0.1-py2-none-any.whl | jq -r '._href')
11+
.. literalinclude:: ../_scripts/upload.sh
12+
:language: bash
1013

11-
Response::
14+
Content GET Response::
1215

1316
{
14-
"_href": "/pulp/api/v3/artifacts/1/",
15-
...
17+
"_artifact": null,
18+
"_created": "2019-07-25T13:57:55.178993Z",
19+
"_href": "/pulp/api/v3/content/python/packages/6172ff0f-3e11-4b5f-8460-bd6a72616747/",
20+
"_type": "python.python",
21+
"author": "",
22+
"author_email": "",
23+
"classifiers": [],
24+
"description": "",
25+
"download_url": "",
26+
"filename": "shelf_reader-0.1-py2-none-any.whl",
27+
"home_page": "",
28+
"keywords": "",
29+
"license": "",
30+
"maintainer": "",
31+
"maintainer_email": "",
32+
"metadata_version": "",
33+
"name": "[]",
34+
"obsoletes_dist": "[]",
35+
"packagetype": "bdist_wheel",
36+
"platform": "",
37+
"project_url": "",
38+
"provides_dist": "[]",
39+
"requires_dist": "[]",
40+
"requires_external": "[]",
41+
"requires_python": "",
42+
"summary": "",
43+
"supported_platform": "",
44+
"version": "0.1"
1645
}
1746

47+
Reference: `Python Content Usage <../restapi.html#tag/content>`_
1848

19-
Reference (pulpcore): `Artifact API Usage
20-
<https://docs.pulpproject.org/en/3.0/nightly/restapi.html#tag/artifacts>`_
21-
22-
Create content from an artifact
23-
-------------------------------
49+
Add content to a repository during one-shot upload
50+
--------------------------------------------------
2451

25-
Now that Pulp has the wheel, its time to make it into a unit of content. The python plugin will
26-
inspect the file and populate its metadata::
52+
One-shot upload can also optionally add the content being created to a repository you specify.
2753

28-
$ http POST $BASE_ADDR/pulp/api/v3/content/python/packages/ _artifact=$ARTIFACT_HREF filename=shelf_reader-0.1-py2-none-any.whl
54+
.. literalinclude:: ../_scripts/upload_with_repo.sh
55+
:language: bash
2956

30-
Response::
57+
Repository GET Response::
3158

3259
{
33-
"_href": "/pulp/api/v3/content/python/packages/1/",
34-
"_artifact": "/pulp/api/v3/artifacts/1/",
35-
"digest": "b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c",
36-
"filename": "shelf_reader-0.1-py2-none-any.whl",
37-
"type": "python"
60+
"_created": "2019-07-25T14:03:48.378437Z",
61+
"_href": "/pulp/api/v3/repositories/135f468f-0c61-4337-9f37-0cd911244bec/versions/1/",
62+
"base_version": null,
63+
"content_summary": {
64+
"added": {
65+
"python.python": {
66+
"count": 1,
67+
"href": "/pulp/api/v3/content/python/packages/?repository_version_added=/pulp/api/v3/repositories/135f468f-0c61-4337-9f37-0cd911244bec/versions/1/"
68+
}
69+
},
70+
"present": {
71+
"python.python": {
72+
"count": 1,
73+
"href": "/pulp/api/v3/content/python/packages/?repository_version=/pulp/api/v3/repositories/135f468f-0c61-4337-9f37-0cd911244bec/versions/1/"
74+
}
75+
},
76+
"removed": {}
77+
},
78+
"number": 1
3879
}
3980

40-
Create a variable for convenience::
41-
42-
$ export CONTENT_HREF=$(http $BASE_ADDR/pulp/api/v3/content/python/packages/ | jq -r '.results[] | select(.filename == "shelf_reader-0.1-py2-none-any.whl") | ._href')
43-
44-
Reference: `Python Content API Usage <../restapi.html#tag/content>`_
4581

46-
Add content to a repository
47-
---------------------------
82+
Reference: `Python Repository Usage <../restapi.html#tag/repositories>`_
4883

49-
See :ref:`add-remove`
84+
For other ways to add content to a repository, see :ref:`add-remove`

‎pulp_python/app/serializers.py

+18
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55
from rest_framework import serializers
66

77
from pulpcore.plugin import models as core_models
8+
from pulpcore.plugin.models import Repository
89
from pulpcore.plugin import serializers as core_serializers
910

1011
from pulp_python.app import models as python_models
@@ -214,6 +215,23 @@ class Meta:
214215
model = python_models.PythonPackageContent
215216

216217

218+
class PythonOneShotUploadSerializer(serializers.Serializer):
219+
"""
220+
A Serializer for PythonOneShotUpload.
221+
"""
222+
223+
repository = serializers.HyperlinkedRelatedField(
224+
help_text=_('A URI of the repository.'),
225+
required=False,
226+
queryset=Repository.objects.all(),
227+
view_name='repositories-detail',
228+
)
229+
file = serializers.FileField(
230+
help_text=_("The python file (i.e. .whl or .tar.gz)."),
231+
required=True,
232+
)
233+
234+
217235
class MinimalPythonPackageContentSerializer(PythonPackageContentSerializer):
218236
"""
219237
A Serializer for PythonPackageContent.

‎pulp_python/app/tasks/upload.py

+80
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
import os
2+
from gettext import gettext as _
3+
import pkginfo
4+
import shutil
5+
import tempfile
6+
7+
from pulpcore.plugin.models import Artifact, CreatedResource, Repository, RepositoryVersion
8+
from rest_framework import serializers
9+
10+
from pulp_python.app.models import PythonPackageContent
11+
from pulp_python.app.utils import parse_project_metadata
12+
13+
14+
DIST_EXTENSIONS = {
15+
".whl": "bdist_wheel",
16+
".exe": "bdist_wininst",
17+
".egg": "bdist_egg",
18+
".tar.bz2": "sdist",
19+
".tar.gz": "sdist",
20+
".zip": "sdist",
21+
}
22+
23+
DIST_TYPES = {
24+
"bdist_wheel": pkginfo.Wheel,
25+
"bdist_wininst": pkginfo.Distribution,
26+
"bdist_egg": pkginfo.BDist,
27+
"sdist": pkginfo.SDist,
28+
}
29+
30+
31+
def one_shot_upload(artifact_pk, filename, repository_pk=None):
32+
"""
33+
One shot upload for pulp_python
34+
35+
Args:
36+
artifact_pk: validated artifact
37+
filename: file name
38+
repository_pk: optional repository to add Content to
39+
"""
40+
# iterate through extensions since splitext does not support things like .tar.gz
41+
for ext, packagetype in DIST_EXTENSIONS.items():
42+
if filename.endswith(ext):
43+
# Copy file to a temp directory under the user provided filename, we do this
44+
# because pkginfo validates that the filename has a valid extension before
45+
# reading it
46+
with tempfile.TemporaryDirectory() as td:
47+
temp_path = os.path.join(td, filename)
48+
artifact = Artifact.objects.get(pk=artifact_pk)
49+
shutil.copy2(artifact.file.path, temp_path)
50+
metadata = DIST_TYPES[packagetype](temp_path)
51+
metadata.packagetype = packagetype
52+
break
53+
else:
54+
raise serializers.ValidationError(_(
55+
"Extension on {} is not a valid python extension "
56+
"(.whl, .exe, .egg, .tar.gz, .tar.bz2, .zip)").format(filename)
57+
)
58+
data = parse_project_metadata(vars(metadata))
59+
data['classifiers'] = [{'name': classifier} for classifier in metadata.classifiers]
60+
data['packagetype'] = metadata.packagetype
61+
data['version'] = metadata.version
62+
data['filename'] = filename
63+
data['_relative_path'] = filename
64+
65+
new_content = PythonPackageContent.objects.create(
66+
filename=filename,
67+
packagetype=metadata.packagetype,
68+
name=data['classifiers'],
69+
version=data['version']
70+
)
71+
72+
queryset = PythonPackageContent.objects.filter(pk=new_content.pk)
73+
74+
if repository_pk:
75+
repository = Repository.objects.get(pk=repository_pk)
76+
with RepositoryVersion.create(repository) as new_version:
77+
new_version.add_content(queryset)
78+
79+
resource = CreatedResource(content_object=new_content)
80+
resource.save()

‎pulp_python/app/urls.py

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
from django.conf.urls import url
2+
3+
from .viewsets import PythonOneShotUploadViewSet
4+
5+
6+
urlpatterns = [
7+
url(r'python/upload/$', PythonOneShotUploadViewSet.as_view({'post': 'create'}))
8+
]

0 commit comments

Comments
 (0)