cc-tracking/tasks/archive/openbanking/DIR-3232/commit.md
2019-05-21 16:40:16 +01:00

602 B

DIR-3232 DEV2- Build an Amazon Linux bootstrapper

Summary

  • Scraper now works inside docker container

  • Scraper uploads to S3 correctly

  • Scraper now set up to use an internal cron job, so starts at predetermined time.

  • Now utilising PM2 to ensure the process restart correctly when the Docke image is rebuilt

  • If the image is restarted during a scrape process that process will restart

  • Dictionary files are now stored in S3

  • If there is poor network connectevitiy then some pages time out. Trying to make the scraper more resillient.

  • Jenkins file created

  • Makefile created tested