The Scrapper App is a lightweight Flask-based web service designed to accept JSON payloads via POST requests to the /v1/submit
endpoint and save them in JSON files formatted with timestamps. It's Dockerized for easy deployment and isolation.
Build the Docker image
docker build -t scrapper .
Run the container
docker run -p 8080:8080 -v $(pwd)/requests:/usr/src/app/requests scrapper
You can now access the service at http://localhost:8080
.
- Method: POST
- Content-Type: application/json
- Body: JSON object
Example:
curl -X POST http://localhost:8080/v1/submit -H "Content-Type: application/json" -d '{"key": "value"}'
Submitted JSON payloads are stored in the requests
directory within the container, with filenames formatted as payload-YYYY-MM-DD-HH:MM:SS,sss.json
.
If you have minimina network with uptime-service-backend, you can replace it with scrapper in order to scrape load to uptime-service-backend.
Change this in docker-compose.yml:
uptime-service-backend-e2e-test:
container_name: uptime-service-backend-e2e-test
- image: 673156464838.dkr.ecr.us-west-2.amazonaws.com/block-producers-uptime:2.0.0rc3-e7ad680-testworld-2-0
+ image: scrapper
volumes:
+ - /home/piotr/scrapper/requests:/usr/src/app/requests
- /home/piotr/.minimina/e2e-test:/local-network
- /home/piotr/.minimina/e2e-test/uptime-storage:/uptime-storage
environment:
and then do:
docker compose down
docker compose create