Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Integrating Django Celery Beat to schedule and execute tasks #119

Merged
merged 41 commits into from
Aug 2, 2021
Merged

Integrating Django Celery Beat to schedule and execute tasks #119

merged 41 commits into from
Aug 2, 2021

Conversation

beshiniii
Copy link
Contributor

@beshiniii beshiniii commented Jul 14, 2021 •
edited by sajithaliyanage
Loading

Description

This PR provides the mechanism to integrate django_celery_beat with the Django backend server.
This provides the following usages:

  • Integrating with Celery beat and workers
  • Integrating with RabbitMQ AMQ broker
  • Execute crawler jobs instantly
  • Schedule crawler jobs as a crontab service
  • Schedule crawler jobs as an interval service
  • Delete schedule crawler jobs
  • Disable schedule crawler jobs
  • Enable environment support for Django and Scrapy apps

Related Issue

#118

beshiniii added 30 commits July 14, 2021 21:59
@beshiniii
refactoring folder structure of crawlerX
058e9c0
@beshiniii
adding celery executor for django
84d92db
@beshiniii
adding sample shared task for celery app
e82f23a
@beshiniii
adding configurations for celery executor with django beats
9ea07bb
@beshiniii
removing existing rabbitmq sender
8a72789
@beshiniii
update requirements.txt file with dependencies
ec73f62
@beshiniii
add and refactoring django service endpoints
c281d11
@beshiniii
refactoring celery configurations
a452c08
@beshiniii
refactoring create new crawl job endpoint functionality
e60717c
@beshiniii
adding function to schedule crawler jobs as interval tasks
6f38a8e
@beshiniii
adding function to schedule crawler jobs as crontab tasks
8594291
@beshiniii
adding function to disable existing scheduled crontab or interval tasks
a88ec04
@beshiniii
adding function to delete existing scheduled crontab or interval tasks
2838016
@beshiniii
removing rabbitMQ broker implementation
71b9892
@beshiniii
removing mongo client implementation from scrapy app
aa4dab5
@beshiniii
adding implemmenation for celery shared task with scrapyd
d45abed
@beshiniii
change master global variable as environment variables
470416c
@beshiniii
change master global variable as environment variables in scrapy app
cf8b176
@beshiniii
add environment variable support for celery task to connect sccrapy api
0fbf97c
@beshiniii
refactor .gitignore file
2101450
@beshiniii
update docker-composer file with celery
3ca7b41
@beshiniii
Refactoring vue application to schedule crawl jobs
5d70d04
@beshiniii
Adding 400 status code for bad requests
6a98f5d
@beshiniii
Refactoring django response format
7b2f7bd
@beshiniii
make crawlerx platform running state with celery beat
6ca105f
@beshiniii
Adding job deleting functionality
be23b8c
@beshiniii
Refactoring job data preview UI
5e3a346
@beshiniii
Update mondogb storing payload when scheduling
50cae04
@beshiniii
Add feature to export data as a JSON
450176f
@beshiniii
Refacroting scrapyd crawler implementation
6e10443
@sajithaliyanage sajithaliyanage merged commit 44ed333 into leopardslab:master Aug 2, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sajithaliyanage sajithaliyanage sajithaliyanage approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@beshiniii @sajithaliyanage