cc-tracking/tasks/archive/openbanking/DIR-3232/Note 2018-11-21T14.20.16.md

Note 2018-11-21T14.20.16

DIR-3232 DEV2- Build an Amazon Linux bootstrapper

• Scraper now works inside docker container

• Scraper uploads to S3 correctly

• Scraper now set up to use an internal cron job, so starts at predetermined time.

• Now utilising PM2 to ensure the process restart correctly when the Docke image is rebuilt

• If the image is restarted during a scrape process that process will restart

• Dictionary files are now stored in S3

**** 2018-11-28 ****

• If there is poor network connectevitiy then some pages time out. Trying to make the scraper more resillient.

• Jenkins file created

• Makefile created tested