28 lines
667 B
Markdown
28 lines
667 B
Markdown
Note 2018-11-21T14.20.16
|
|
========================
|
|
|
|
DIR-3232 DEV2- Build an Amazon Linux bootstrapper
|
|
|
|
* Scraper now works inside docker container
|
|
|
|
* Scraper uploads to S3 correctly
|
|
|
|
* Scraper now set up to use an internal cron job, so starts at predetermined time.
|
|
|
|
* Now utilising PM2 to ensure the process restart correctly when the Docke image is rebuilt
|
|
|
|
* If the image is restarted during a scrape process that process will restart
|
|
|
|
* Dictionary files are now stored in S3
|
|
|
|
|
|
|
|
**** 2018-11-28 ****
|
|
|
|
* If there is poor network connectevitiy then some pages time out. Trying to make the scraper more resillient.
|
|
|
|
* Jenkins file created
|
|
|
|
* Makefile created tested
|
|
|