22 lines
602 B
Markdown
22 lines
602 B
Markdown
|
DIR-3232 DEV2- Build an Amazon Linux bootstrapper
|
||
|
|
|
||
|
|
# Summary
|
||
|
|
|
||
|
|
* Scraper now works inside docker container
|
||
|
|
|
||
|
|
* Scraper uploads to S3 correctly
|
||
|
|
|
||
|
|
* Scraper now set up to use an internal cron job, so starts at predetermined time.
|
||
|
|
|
||
|
|
* Now utilising PM2 to ensure the process restart correctly when the Docke image is rebuilt
|
||
|
|
|
||
|
|
* If the image is restarted during a scrape process that process will restart
|
||
|
|
|
||
|
|
* Dictionary files are now stored in S3
|
||
|
|
|
||
|
|
* If there is poor network connectevitiy then some pages time out. Trying to make the scraper more resillient.
|
||
|
|
|
||
|
|
* Jenkins file created
|
||
|
|
|
||
|
|
* Makefile created tested
|