Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add admin command for scraping new item #1

Open
3 of 4 tasks
maciejmatczak opened this issue Nov 18, 2019 · 2 comments
Open
3 of 4 tasks

Add admin command for scraping new item #1

maciejmatczak opened this issue Nov 18, 2019 · 2 comments
Assignees
Labels
enhancement New feature or request

Comments

@maciejmatczak
Copy link
Owner

maciejmatczak commented Nov 18, 2019

  • job should be run once every 24 hours - around time chosen by the user, but the command will be invoked every 5 (?) minutes
  • optional random delay as command input
  • new item should be added only when it is not yet registered (according to known facts - the same url should produce new item register if any of the gathered parameter is changed)
  • new items should be highlighted as "new" - up to the moment when user views them
  • command to be installed by cron

Task list:

  • implement initial command which creates the new items on every active job on runtime (no delay yet)
  • cover it with small tests
  • add mechanism for running every 5 minutes, but invoking scraping when around specific hour
  • cover with functional tests
@maciejmatczak maciejmatczak added the enhancement New feature or request label Nov 18, 2019
@maciejmatczak maciejmatczak self-assigned this Nov 18, 2019
@maciejmatczak
Copy link
Owner Author

maciejmatczak commented Nov 18, 2019

Mechanism to run every 5 minutes
Job needs to be run in one of the 5 minutes window during a day, so:

  • add new model field, daily_run; if set time is smaller then current time and the job was not yet run - execute it and change the status
  • every 24 hours reset all jobs

Easy to implement. Can have issues around reset time. Maybe there are better ready solutions (preferably light) for cron like tasks?

@maciejmatczak
Copy link
Owner Author

Something more user friendly options would be better, like every 5 minutes, 1 hour, 24 hours, etc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant