update 2019-03-21

qnotes/Ascii Emojis.md

@@ -0,0 +1,331 @@
+Ascii Emojis
+==========
+
+```
+ʘ‿ʘ
+innocent face 
+
+ಠ_ಠ
+reddit disapproval face 
+
+(╯°□°）╯︵ ┻━┻
+table flip / flipping table 
+
+┬─┬ ノ( ゜-゜ノ)
+put the table back
+
+┬─┬⃰͡ (ᵔᵕᵔ͜ )
+tidy up / dust the table
+
+┻━┻ ︵ヽ(`Д´)ﾉ︵ ┻━┻
+double Flip / double angry
+
+ლ(｀ー´ლ)
+fisticuffs
+
+ʕ•ᴥ•ʔ
+cute bear 
+
+ʕᵔᴥᵔʔ
+squinting bear
+
+ʕ •`ᴥ•´ʔ
+GTFO Bear
+
+(｡◕‿◕｡)
+cute face with big eyes 
+
+（　ﾟДﾟ）
+surprised / loudmouthed 
+
+¯\_(ツ)_/¯
+shrug face  
+
+¯\(°_o)/¯
+meh
+
+(`･ω･´)
+feel perky  
+
+(╬ ಠ益ಠ)
+angry face
+
+☜(⌒▽⌒)☞
+excited 
+
+ε=ε=ε=┌(;*´Д`)ﾉ
+running 
+
+ヽ(´▽`)/
+happy face  
+
+ヽ(´ー｀)ノ
+basking in glory  
+
+ᵒᴥᵒ#
+kitty emote
+
+V•ᴥ•V
+fido
+
+ฅ^•ﻌ•^ฅ
+meow
+
+（ ^_^）o自自o（^_^ ）
+cheers  
+
+ಠ‿ಠ
+devious smile
+
+( ͡° ͜ʖ ͡°)
+4chan emoticon  
+
+ಥ_ಥ
+crying face 
+
+ಥ﹏ಥ
+breakdown
+
+٩◔̯◔۶
+disagree
+
+ᕙ(⇀‸↼‶)ᕗ
+flexing 
+
+ᕦ(ò_óˇ)ᕤ
+do you even lift bro?
+
+⊂(◉‿◉)つ
+kirby
+
+q(❂‿❂)p
+tripping out  
+
+⊙﹏⊙
+discombobulated 
+
+¯\_(⊙︿⊙)_/¯
+sad and confused  
+
+°‿‿°
+japanese lion face
+
+¿ⓧ_ⓧﮌ
+confused  
+
+(⊙.☉)7
+confused scratch
+
+(´･_･`)
+worried
+
+щ（ﾟДﾟщ）
+dear god why  
+
+٩(๏_๏)۶
+staring 
+
+ఠ_ఠ
+pretty eyes
+
+ᕕ( ᐛ )ᕗ
+strut
+
+(⊙_◎)
+zoned
+
+ミ●﹏☉ミ
+crazy
+
+༼∵༽ ༼⍨༽ ༼⍢༽ ༼⍤༽
+trolling
+
+ヽ༼ ಠ益ಠ ༽ﾉ
+angry troll
+
+t(-_-t)
+fuck it
+
+(ಥ⌣ಥ)
+sad face
+
+(づ￣ ³￣)づ
+hugger
+
+(づ｡◕‿‿◕｡)づ
+stranger danger
+
+(ノಠ ∩ಠ)ノ彡( \o°o)\
+flip friend
+
+｡ﾟ( ﾟஇ‸இﾟ)ﾟ｡
+cry face
+
+༼ ༎ຶ ෴ ༎ຶ༽
+cry troll
+
+“ヽ(´▽｀)ノ”
+TGIF
+
+┌(ㆆ㉨ㆆ)ʃ
+dancing 
+
+눈_눈
+sleepy
+
+( ఠൠఠ )ﾉ
+angry birds
+
+乁( ◔ ౪◔)「      ┑(￣Д ￣)┍
+no support 
+
+(๑•́ ₃ •̀๑) 
+shy 
+
+⁽⁽ଘ( ˊᵕˋ )ଓ⁾⁾
+fly away
+
+◔_◔
+careless
+
+♥‿♥
+love
+
+ԅ(≖‿≖ԅ)
+Touchy Feely
+
+( ˘ ³˘)♥ 
+Kissing
+
+( ˇ෴ˇ )
+shark face / jagged mustache
+
+ヾ(-_- )ゞ
+emo dance 
+
+♪♪ ヽ(ˇ∀ˇ )ゞ
+dance 
+
+ヾ(´〇`)ﾉ♪♪♪
+opera
+
+ʕ •́؈•̀ ₎
+winnie the pooh
+
+ʕ •́؈•̀)
+winnie the pooh
+
+ლ(•́•́ლ)
+boxing
+
+(ง'̀-'́)ง
+fight
+
+◖ᵔᴥᵔ◗ ♪ ♫ 
+listening to headphones
+
+{•̃_•̃}
+robot
+
+(ᵔᴥᵔ)
+seal
+
+(Ծ‸ Ծ)
+questionable / dislike
+
+(•̀ᴗ•́)و ̑̑
+Winning!
+
+[¬º-°]¬
+Zombie
+
+(☞ﾟヮﾟ)☞
+pointing
+
+''⌐(ಠ۾ಠ)¬'''
+chasing / running away
+
+(っ•́｡•́)♪♬
+whistling / music
+
+(҂◡_◡) 
+injured
+
+ƪ(ړײ)‎ƪ​​
+creeper
+
+⥀.⥀
+eye roll
+
+ح˚௰˚づ 
+flying
+
+♨_♨
+things that can't be unseen
+
+(._.)
+looking down
+
+(⊃｡•́‿•̀｡)⊃
+im a hugger
+
+(∩｀-´)⊃━☆ﾟ.*･｡ﾟ
+wizard
+
+(っ˘ڡ˘ς)
+yum
+
+( ఠ ͟ʖ ఠ)
+judging
+
+( ͡ಠ ʖ̯ ͡ಠ)
+tired / older
+
+( ಠ ʖ̯ ಠ)
+dislike / depressed
+
+(งツ)ว
+hitchhiking
+
+(◠﹏◠)
+satisfied
+
+(ᵟຶ︵ ᵟຶ)
+sad and crying
+
+(っ▀¯▀)つ
+stunna shades
+
+ʚ(•｀
+chicken
+
+(´ж｀ς)
+barf
+
+(° ͜ʖ͡°)╭∩╮
+fuck off
+
+ʕʘ̅͜ʘ̅ʔ
+smiley toast
+
+ح(•̀ж•́)ง † 
+exorcism
+
+-`ღ´-
+love
+
+(⩾﹏⩽)
+taking a dump / straining
+
+ヽ( •_)ᕗ
+dab
+
+~(^-^)~
+Wave Dance
+
+\(ᵔᵕᵔ)/
+Happy Hug
+```
+
+Note:
+The emojis above consist of Unicode characters and not ASCII characters. I found that people often search for ASCII over unicode not understanding the difference so I've kept the repo name intact for SEO reasons. See https://en.wikipedia.org/wiki/ASCII v. https://en.wikipedia.org/wiki/Unicode

qnotes/Commit template.md

@@ -0,0 +1,61 @@
+# Commit template
+
+
+BRIDGE-1163 - Add description to rule
+
+# Summary
+
+* An orderDescription field was added to single-rule-create in Swagger.
+* Marked payerInstrument and payeeID as required fields in single-rule-create
+* Updated /specs/create-rule.e2e.spec.js to test for orderDescription & missing payerInstrument and payeeID
+
+# Test Plan
+
+* Run All tests
+* Run E2E: Create a rule
+
+```
+E2E: Create a rule PASSED
+    Tests the body PASSED
+        Tests with missing required parameters PASSED
+            
+            Missing paymentInstrument contents PASSED
+                ✓ Test with no payer instrument ID parameter PASSED
+                ✓ Test with no payer instrument key parameter PASSED
+
+            Missing Trigger PASSED
+                ✓ Test with no Trigger parameter PASSED
+                
+            Missing Payment rules PASSED
+                ✓ Test with no Payment rules parameter PASSED
+            
+            Missing Rule parameters PASSED
+                ✓ Test with no rule payerInstrument parameter PASSED
+                ✓ Test with no rule payeeID parameter PASSED
+
+    Bad data tests PASSED
+        ✓ Test with a badly formatted rule payer instrument ID parameter PASSED
+        ✓ Test with a badly formatted rule payer key parameter PASSED
+        ✓ Test with a badly formatted rule payeeID parameter PASSED
+        ✓ Test with a short rule payeeID parameter PASSED
+        ✓ Test with a long rule payeeID parameter PASSED
+        ✓ Test with a badly formatted rule percentageShare parameter PASSED
+        ✓ Test with a out of range rule percentageShare parameter PASSED
+        ✓ Test with a badly formatted Trigger Stage parameter PASSED
+        ✓ Test with a badly formatted Trigger Type parameter PASSED
+        ✓ Test with a long rule description parameter PASSED
+
+    Good parameter data tests PASSED
+        ✓ Test with a full set of correct parameters PASSED
+
+    Authorization PASSED
+        ✓ Test rejects invalid bearer token PASSED
+
+    Tests the response PASSED
+        Response validation for 201 Created responses PASSED
+            ✓ Test returns a response with `valid` ID PASSED
+            ✓ Test rejects a response with `invalid` ID PASSED
+            ✓ Test rejects an empty response PASSED
+            
+21 Of 21 Passing
+```

qnotes/Note 2018-11-06T13.08.43.md

@@ -0,0 +1,9 @@
+Note 2018-11-06T13.08.43
+========================
+
+/var/azuracast/stations/rakh_sleep/media
+
+/mnt/dh3/docker/stations/rakh_sleep/media
+
+
+

qnotes/Note 2018-11-14T10.44.21.md

@@ -0,0 +1,18 @@
+Note 2018-11-14T10.44.21
+========================
+
+Mercedes Garcia <mercedes.garcia@malikshaw.com>
+consent@projectpeople.com
+liam.oneill@talentinternational.com
+david.arkinstall@g2recruitment.com
+kyle.thomas@paratuspeople.com
+david.dowdall@evolution-contract.co.uk
+Mohsin.Miah@squareoneresources.com
+Fiona.Macleod@HarveyNash.com
+Chris.Yardley@search.co.uk
+kieron.mackerness@randstad.co.uk
+euan.wilson@plexusrs.com
+privacy@modis.co.uk
+ian.fitzgerald@evolutionjobs.co.uk
+M.Freer@roc-search.com
+group@hays-uk.com

qnotes/Note 2018-11-14T17.20.43.md

@@ -0,0 +1,7 @@
+Note 2018-11-14T17.20.43
+========================
+
+not too sure what you're on about i think its quite a nice keyboard tbf
+
+
+his is a thing and im going to test the typig

qnotes/Note 2019-01-22T13.37.27.md

@@ -0,0 +1,19 @@
+Interviews
+========================
+
+
+Some of the data we process is in a mix of cs and pdf, how would you deal with processing that.
+
+We deal with a lot of data from different sources in different spoken languages, how would you normalise this data?
+
+
+
+The data we get can be quite raw and messy, how would you ensure that it as accurate and useable?
+
+4) Mention what is data cleansing?
+
+Data cleaning also referred as data cleansing, deals with identifying and removing errors and inconsistencies from data in order to enhance the quality of data.
+
+
+
+

qnotes/Tims Tweets.md

@@ -0,0 +1,23 @@
+Tims Tweets
+========================
+
+```
+
+ʕ•ᴥ•ʔ
+
+```
+
+
+Happy Pickle day..  🐻❤️🥒 #littlebear #lovepickles
+
+
+Yay!! it's Have a Party with Your Bear Day! One of the best times of the year. Give your special bear a hug for me xx 🐻❤️🐻 #HaveaPartywithYourBearDay #littlebear #hugabear
+
+❤️
+
+Happy Halloween!
+🎃🦇☠️👻🐻 #HappyHalloween #littlebear #spookybear
+
+🏴󠁧󠁢󠁳󠁣 scotland flag
+
+🇸🇪󠁴󠁿

qnotes/docker.md

@@ -0,0 +1,182 @@
+docker
+======
+
+
+
+# Docker #
+
+To install PostgreSQL, Postgres Admin console, Redis and the test Web app, you would use docker-compose.
+
+### Install Docker ###
+
+Download and install Docker for Mac : [Docker Community Edition for Mac](https://store.docker.com/editions/community/docker-ce-desktop-mac)
+
+Docker for Mac already includes Compose along with other Docker apps, so Mac users do not need to install Compose separately.
+
+### Run Docker compose ###
+
+docker-compose up -d
+
+
+### Configure components ###
+
+Looking at the docker-compose.yml file section by section:
+
+#### PostgreSQL ####
+
+```
+  PostgreSQL:
+    image: postgres:alpine
+    ports:
+      - 5432:5432
+    restart: always
+    env_file: .env
+    volumes:
+      - "$ROOT/postgresql:/var/lib/postgresql/data"
+      - ./setup/postgres/init.sql:/docker-entrypoint-initdb.d/init.sql
+```
+
+This sets PostgreSQL to use the Postgres:Alpine image. This is a version of the Postgres image built upon the Alpine Linux disk image, it is 5MB so makes for a slimmer image.
+
+A 'friendly' name is set for the image, otherwise it will create the running image with a slightly cryptic name which appears to be based upon the current directory, the name of the container and an integer. Here, the container name is set to PostgreSQL
+
+It then exposes the Postgres communication port to 5432. This is the standard port for Postgres.
+
+The restart option is set to `always` to ensure the image is started again if the host is rebooted. There are other options available, these options can be found here: [Docker-compose:Restart](https://docs.docker.com/compose/compose-file/#restart)
+
+Then the environment file is set by pointing `env_file` to `.env`. This allows the use of predefined variables instead of editing the docker-compose file to set things such as volumes. 
+
+The Postgres image utilises a number of environment variables to set user, password and server details. these values are contained within the `.env` file as `POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_DB`. They can be changed within the `.env` file.
+
+Finally, the volumes are set. 
+
+The first one: 
+
+```
+"$ROOT/postgresql:/var/lib/postgresql/data"
+```
+
+This makes the Postgres image persist data in a local folder `./postgressql`. Doing this prevents data from being lost if the image is stopped. However, if `docker-compose down -v` is run then the persistant data will be removed.
+
+The `$ROOT` value is stored in the environment file, and can be configured there. 
+
+The second volume:
+
+```
+./setup/postgres/init.sql:/docker-entrypoint-initdb.d/init.sql
+```
+
+This is used to recreate the Postgres database from scratch in the case of starting everything with `./docker-compose up`. 
+
+The official Postgres docker image will run .sql scripts found in the /docker-entrypoint-initdb.d/ folder. So that is used by pointing `/docker-entrypoint-initdb.d/init.sql` to a local file.
+
+The Postgres image can be stopped and restarted again individually. Since the container name was set in the configuration, the following can be applied:
+
+```
+docker stop PostgreSQL
+
+docker restart PostgreSQL
+```
+
+#### Adminer ####
+
+Adminer is a database management console.
+
+```
+adminer:
+    image: adminer
+    container_name: adminer
+    restart: always
+    depends_on:
+      - PostgreSQL
+    ports:
+      - 8080:8080
+```
+
+The image is set to use the adminer image, and the container name is set to be `adminer` instead of the automatically generated container name.
+
+Again, the restart option is set to `always` so that adminer will restart on power loss.
+
+`Depends_on` is set to `PostgreSQL` since this is the management console for it. 
+
+The port `8080` is then exposed allowing users to access it. It is typically accessed via [`http://localhost:8080`](http://localhost:8080)
+
+Since everything is being launched by docker-compose, there is a background network created that allows the services to communicated with each other. Each service takes the interface name from the configuration file. So the Postgres server will be called PostgreSQL.
+
+When logging in via the Adminer console, the System should be set to Postgres and the Server should be set to PostgresSQL
+
+The username, password and database to connect to can be found within the `.env` file as `POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_DB`.
+
+#### Redis ####
+
+Redis is an open source key-value store that functions as a data structure server.
+
+```
+redis:
+    image: redis:alpine
+    container_name: redis
+    restart: always
+    volumes:
+      - "$ROOT/redis:/data"
+      
+```  
+
+The image is set to use the lightweight redis:alpine image, and the container name is set to be `redis` instead of the automatically generated container name.
+
+Again, the restart option is set to `always` so that adminer will restart on power loss.
+
+The persistent data volume is set to be `./redis` based by using the ROOT value from the `.env` file.
+
+A blank configuration file is included in the `./setup/redist` folder. If this is required to configure redis, then an additional volume should be added to the image. Such as:
+
+```
+redis:
+    image: redis:alpine
+    container_name: redis
+    restart: always
+    volumes:
+      - "$ROOT/redis:/data"
+      - "./setup/redis/redis.conf:/usr/local/etc/redis/redis.conf"
+```  
+
+#### Web ####
+
+This is a small NodeJS based server, every 15 seconds it insert data into the database. It also serves a small single page app which retrieves the data from the server and displays a list of entries.
+
+It is an example of how to use Docker Compose to build an app from the source code then serve it.
+
+```
+web:
+    build: ./setup/web/.
+    container_name: web
+    command: npm run start
+    volumes:
+      - /usr/app/node_modules
+    ports:
+      - 9000:9000
+    depends_on:
+      - PostgreSQL
+      - redis
+    environment:
+      - HOST=${HOST}
+      - DATABASE=${POSTGRES_DB}
+      - USER=${POSTGRES_USER}
+      - PASSWORD=${POSTGRES_PASSWORD}
+      - DATABASE_URL=${DATABASE_URL}
+```
+
+There is no offical image for this, as it builds a new 'image' using the Dockerfile in ./setup/web.
+
+This Dockerfile creates a new Node container, copies all the files in ./setup/web into it then runs npm install. This creates the base image for the container.
+
+The container name is set to be `web` instead of the automatically generated container name.
+
+The command to run once the image has been built is `npm run start`. This launches a NPM script inside the package.json file copied to the container.
+
+A persistent volume is created for the node app on `/user/app`.
+
+`Depends_on` is set to `PostgreSQL` and `redis` as these are required to be installed before this app can be run.
+
+The database connection information is sourced from the .env file and passed to the Node app via environment variables.
+
+When the web app is finally running it can be available from [`http://localhost:9000`](http://localhost:9000)

qnotes/log.md

@@ -0,0 +1,319 @@
+log
+===
+
+
+
+## Log ##
+
+
+##### 2018-03-19 // Monday #####
+
+**[BRIDGE_592](https://comcarde.atlassian.net/browse/BRIDGE-592)** Use validated `req.swagger.params.body.value` not unvalidated `req.body` in controllers
+
+##### 2018-03-20 // Tuesday #####
+
+Continuing and finishing BRIDGE_592
+
+##### 2018-03-21 // Wednesday #####
+
+Started working on BRIDGE_588: Updating e2e tests using the new testing helpers.
+
+##### 2018-03-22 // Thursday #####
+
+Continuing and finishing BRIDGE_588
+
+updated pay-to-stored-worldpay-account.e2e.spec.js to use the new testing helper.
+
+Tests finally past
+
+##### 2018-03-23 // Friday #####
+
+Started looking at BRIDGE-619, the main story for generating aa receive code.
+
+Creating swagger end point for receivecode
+
+Creating tests for receivecode
+
+---
+
+
+##### 2018-03-26 // Monday #####
+Fixed a few issues with receivecodes stub.
+Returning the 502 code required {info, code}
+Recommitted, and merged
+
+##### 2018-03-27 // Tuesday #####
+BRIDGE-641 Make sure everything works as expected
+
+Worked through the acceptance criteria and documented all results.
+
+##### 2018-03-28 // Wednesday #####
+
+BRIDGE-641 - Redo acceptance critereia tests on dev?
+
+BRIDGE-646 - Start Implement for Bridge-643 'As a User I want to be able to redeem a receivecode with a new credit/debit card'
+
+Started writing redeem.spec.js 
+
+##### 2018-03-29 // Thursday #####
+
+BRIDGE-646 - Continuing with this item, finishing the tests.
+
+*DONE* BRIDGE-641 - Redo acceptance criteria tests on dev. I have login and db details now.
+
+##### 2018-03-30 // Friday #####
+
+BRIDGE-646 - Starting E2E tests for using a receivecode with a new credit/debit card, expanding other tests
+
+---
+
+
+##### 2018-04-02 // Monday #####
+
+BRIDGE-646 - Plugged my code into the work by John from BRIDGE-644
+
+Updated tests to work with this merged code
+
+Updated code to work with the new renaming Expiry work from Richard
+
+##### 2018-04-03 // Tuesday #####
+
+*DONE* Finally merged the code for BRIDGE-646. Task moved to done
+
+*DONE* Picked up BRIDGE-647 Make sure everything works as expected
+
+*DONE* Picked up BRIDGE-749 No compliance log entry for this story, updated confluence [https://comcarde.atlassian.net/wiki/spaces/TA/pages/96108550/Make+a+Worldpay+payment+using+a+receivecode+and+Payment+Instrument+Payment+Details+POST+payments+receivecode]
+
+##### 2018-04-04 // Wednesday #####
+
+Sprint planning
+
+##### 2018-04-05 // Thursday #####
+
+Writing user story and tasks for Cancelling RTPs - BRIDGE-779
+
+Started BRIDGE-760 - Swagger definition
+
+##### 2018-04-06 // Friday #####
+
+*DONE* BRIDGE-760 - Swagger definition
+Started BRIDGE-764 - E2E Test of swagger spec
+
+---
+
+##### 2018-04-09 // Monday #####
+
+BRIDGE-764 - E2E Test of swagger spec
+Started BRIDGE-796 - Create a robust getByQuery
+
+##### 2018-04-10 // Tuesday #####
+
+*DONE* Continuing with BRIDGE-796
+
+*DONE* Started BRIDGE-798 - Swagger response example values do not match actual
+
+Continued BRIDGE-764 now that BRIDGE0796, BRIDGE-798 are complete
+
+##### 2018-04-11 // Wednesday #####
+
+*DONE* BRIDGE-764 - E2E tests for RPT list
+
+Created story and tasks for BRIDGE-813 - As a system, I want my receivables objects to be identified as such
+
+Created story and tasks for BRIDGE-867 - As an operator, I want to list all ATPs related to my receivable RTP
+
+Created story and tasks for BRIDGE-857 - As a user I want to list all ATPs related to my payable RTP
+
+##### 2018-04-12 // Thursday #####
+
+*DONE* Started BRIDGE-800 - Swagger def
+
+Started BRIDGE-801 - E2E Tests of Swagger Def
+
+##### 2018-04-13 // Friday #####
+
+*DONE* BRIDGE-801 - E2E Tests of Swagger Def
+
+Started BRIDGE-858 - Update swagger def
+
+
+---
+
+##### 2018-04-16 // Monday #####
+
+*DONE* BRIDGE-858 - Update swagger def
+
+
+##### 2018-04-17 // Tuesday #####
+
+Started BRIDGE-860 - E2E tests for swagger
+
+*DONE* Started BRIDGE-885 - Swagger UI for list returns a single ATP.
+
+
+##### 2018-04-18 // Wednesday #####
+
+Sprint planning
+
+BRIDGE-885 - Swagger UI for list returns a single ATP.
+
+*DONE* Started BRIDGE-837 - Update swagger def
+
+Fixed small issue with BRIDGE-868
+
+##### 2018-04-19 // Thursday #####
+
+Started BRIDGE-838 - E2E tests for swagger
+
+Started BRIDGE-844 - Update swagger def
+
+##### 2018-04-20 // Friday #####
+
+*DONE* BRIDGE-838 - E2E tests for swagger
+
+Started BRIDGE-838 - Implementation
+
+Started BRIDGE-822 - As a user I want to pay an RTP in full (using stored or new instrument)
+
+---
+
+##### 2018-04-23 // Monday #####
+
+Ill
+
+##### 2018-04-24 // Tuesday #####
+
+*DONE* BRIDGE-860 - e2e test swagger
+
+*DONE* Started BRIDGE-877 - Spelling mistake in "Merhcant" error message
+
+*DONE* Started BRIDGE-901 - Unit test decrypt-card.spec.js sometimes fails due to MS difference
+
+##### 2018-04-25 // Wednesday #####
+
+Started BRODGE-895 - INVESTIGATION: check if TTL query deletions can be ignored based on another property
+
+##### 2018-04-26 // Thursday #####
+
+Ill
+
+##### 2018-04-27 // Friday #####
+
+Ill
+
+---
+
+##### 2018-04-30 // Monday #####
+
+Started BRIDGE-922 - Swagger E2E tests
+
+##### 2018-05-01 // Tuesday #####
+
+*DONE* BRIDGE-922 - Swagger E2E tests
+
+##### 2018-05-02 // Wednesday #####
+
+Sprint Planning
+
+Started BRIDGE-1001 - processorResponse inconsistent location
+
+##### 2018-04-03 // Thursday #####
+
+BRIDGE-1001 - processorResponse inconsistent location
+
+##### 2018-04-04 // Friday #####
+
+*DONE* BRIDGE-1001 - processorResponse inconsistent location
+
+Started BRIDGE-923 - Implementation
+
+Started BRIDGE-1027 processorResponse not being populated on error
+
+---
+
+##### 2018-05-07 // Monday #####
+
+*DONE* BRIDGE-1027 processorResponse not being populated on error
+
+##### 2018-05-08 // Tuesday #####
+
+Started BRIDGE-923 - Implementation
+
+##### 2018-05-09 // Wednesday #####
+
+BRIDGE-923 - Implementation
+
+##### 2018-05-10 // Thursday #####
+
+Commited BRIDGE-923 - Implementation
+
+##### 2018-05-11 // Friday #####
+
+Holiday
+
+---
+
+##### 2018-05-15 // Monday #####
+
+Holiday
+
+##### 2018-05-16 // Tuesday #####
+
+Started BRIDGE-970 - E2E Swagger tests
+
+##### 2018-05-17 // Wednesday #####
+
+Sprint planning
+
+BRIDGE-973 - Make sure it works
+
+##### 2018-05-18 // Thursday #####
+
+BRIDGE-973 - Make sure it work
+
+Started THX-1138 - Rule can be confirmed with different user / different payee instrument
+
+##### 2018-05-19 // Friday #####
+
+*DONE* BRIDGE-1138 - Rule can be confirmed with different user / different payee instrument
+
+*DONE* BRIDGE-973 - Make sure it work
+
+---
+
+##### 2018-05-21 // Monday #####
+
+BRIDGE-1162 - metaData appears to be RulesTriggerData
+
+Started BRIDGE-1163 - Add description to rule
+
+##### 2018-05-22 // Tuesday #####
+
+*DONE* BRIDGE-1162 - metaData appears to be RulesTriggerData
+
+*DONE* BRIDGE-1163 - Add description to rule
+
+##### 2018-05-23 // Wednesday #####
+
+*DONE* BRIDGE-1165 - Wiki not updated
+
+Started BRIDGE-906 Request Id should implemented in the Activity Log
+
+##### 2018-05-24 // Thursday #####
+
+BRIDGE-906 Request Id should implemented in the Activity Log
+
+##### 2018-05-25 // Friday #####
+
+*DONE* BRIDGE-906 Request Id should implemented in the Activity Log
+
+
+---
+
+##### 2018-05-28 // Monday #####
+
+Working with Docker to build services for Postgres, Redis and some other containers
+
+##### 2018-05-29 // Tuesday #####
+
+Continuing with the docker work

tasks/Blocked ips.md

@@ -0,0 +1,77 @@
+blocked ips
+========================
+
+ 45.125.65.88
+141.98.10.37
+193.169.254.67
+185.234.217.188
+45.125.65.65
+
+141.98.80.15
+
+ 45.125.65.63
+
+185.234.217.188
+
+45.125.66.183
+
+185.53.91.24
+185.53.91.24
+
+
+sudo iptables -A INPUT -s 45.125.65.0/24 -j DROP
+
+sudo iptables -A INPUT -s 141.98.10.0/24 -j DROP
+
+sudo iptables -A INPUT -s 193.169.254.0/24 -j DROP
+
+sudo iptables -A INPUT -s 185.234.217.0/24 -j DROP
+
+sudo iptables -A INPUT -s 185.36.81.0/24 -j DROP
+
+
+
+sudo iptables -A INPUT -s 141.98.80.0/24 -j DROP
+
+sudo iptables -A INPUT -s 45.125.65.0/24 -j DROP
+
+sudo iptables -A INPUT -s 185.234.217.188 -j DROP
+
+sudo iptables -A INPUT -s 45.125.66.0/24 -j DROP
+
+sudo iptables -A INPUT -s 185.53.91.24 -j DROP
+
+https://help.ubuntu.com/community/PortKnocking
+
+```
+
+To block 116.10.191.* addresses:
+
+$ sudo iptables -A INPUT -s 116.10.191.0/24 -j DROP
+
+To block 116.10.*.* addresses:
+
+$ sudo iptables -A INPUT -s 116.10.0.0/16 -j DROP
+
+To block 116.*.*.* addresses:
+
+$ sudo iptables -A INPUT -s 116.0.0.0/8 -j DROP
+
+But be careful what you block using this method. You don't want to prevent legitmate traffic from reaching the host.
+
+edit: as pointed out, iptables evaluates rules in sequential order. Rules higher in the ruleset are applied before rules lower in the ruleset. So if there's a rule higher in your ruleset that allows said traffic, then appending (iptables -A) the DROP rule will not produce the intended blocking result. In this case, insert (iptables -I) the rule either:
+
+as the first rule
+sudo iptables -I ...
+
+or before the allow rule
+sudo iptables --line-numbers -vnL
+
+say that shows rule number 3 allows ssh traffic and you want to block ssh for an ip range. -I takes an argument of an integer that's the location in your ruleset you want the new rule to be inserted
+
+iptables -I 2 ...
+
+
+
+ckiGO99l
+```

tasks/DIR-3059/DIR-3059 2018-11-06T16_16_17.md

@@ -0,0 +1,9 @@
+DIR-3059
+========================
+
+Extend OBDFCASCRAPE to be able to download Cypriot NCA Credit Institute data
+
+https://www.centralbank.cy/en/licensing-supervision/banks/register-of-credit-institutions-operating-in-cyprus
+
+
+

tasks/DIR-3059/commit.md

@@ -0,0 +1,45 @@
+DIR-3059 Extend OBDFCASCRAPE to be able to download Cypriot NCA Credit Institute data
+
+# Summary
+
+* Implement Credit Institute scraping
+* Extended test to test Credit Institute scraping
+
+# Test Plan
+
+* Run All tests
+
+```
+# CYPRUS:: Scrape a Credit Service
+# CYPRUS::Extract Local Credit Institutions from Page
+ok 1 Extracted Local Credit Institutions from Page
+# CYPRUS::Extract Foreign Credit Institutions from Page
+ok 2 Extracted Foreign Credit Institutions from Page
+# CYPRUS::Extract ALL Credit Institutions from Page
+[2018-11-06T16:25:36.859] [INFO] CY - Credit institutes
+[2018-11-06T16:25:36.867] [INFO] Scraper - fullPath tests/sink/201811061625
+[2018-11-06T16:25:36.871] [DEBUG] Scraper - Snapshot tests/sink/201811061625/creditInstitutes.png
+ok 3 Extracted ALL Credit Institutions from Page
+
+1..3
+# tests 3
+# pass  3
+
+# ok
+
+# CYPRUS::WHOIS Test
+CMD dig www.centralbank.cy +time=3 +retry=1
+ok 1 Get Raw CY WhoIS
+ok 2 No JSON for Cyprus
+ok 3 Get JSON WhoIS for IP Address
+# CYPRUS::SSL Test
+[2018-11-06T16:19:50.502] [DEBUG] Scraper - Cert for: www.centralbank.cy
+ok 4 Get SSL Certificate
+
+1..4
+# tests 4
+# pass  4
+
+# ok
+
+```

tasks/DIR-3089/DIR-3089.md

@@ -0,0 +1,10 @@
+DIR-3089
+========================
+Extend OBDFCASCRAPE to be able to upload Irish NCA data
+
+* Added archiving
+
+* Added uploading to S3
+
+* Removed timestamping of folders within the archive to make it easier for Ben to process
+

tasks/DIR-3089/commit.md

@@ -0,0 +1,10 @@
+DIR-3089 Extend OBDFCASCRAPE to be able to upload Irish NCA data
+
+# Summary
+
+* Added archiving - The contents of the artefact folder is compressed
+
+* Added uploading to S3
+
+* Removed timestamping of folders within the archive to make it easier for Ben to process
+

tasks/DIR-3096/Note 2018-11-08T11.05.01.md

@@ -0,0 +1,18 @@
+DIR-3096
+========================
+
+Extend OBDFCASCRAPE to be able to upload Cyprus NCA data
+
+
+
+ OBDFCASCRAPE extended to enable archiving of the data scraped from http://www.mof.gov.cy/mof/mof.nsf/index_en/index_en?OpenDocument , https://www.cysec.gov.cy/en-GB/home/ , http://www.centralbank.gov.cy/nqcontent.cfm?a_id=1&lang=en
+ non-repudiation data for each downloaded file stored in the Artefact Store using path */artefacts/IE/CBOI/YYYYMMDDHHMM/filename-nr.zip* (see https://openbanking.atlassian.net/wiki/spaces/DEV/pages/465634191/DRAFT+NCA+Data+Acquisition+and+Access+Design+Document)
+ Archived data uploaded to the correct S3 store
+ unit tests included with the source code of the agent
+
+
+
+
+
+
+As a user interested in the Cypriot NCA PSD2 data, I would like to data including non-repudiation data which has been retrieved from Cyprus to be uploaded to the artefact Store.

tasks/DIR-3096/commit.md

@@ -0,0 +1,9 @@
+DIR-3096 Extend OBDFCASCRAPE to be able to upload Cyprus NCA data
+
+# Summary
+
+* Added archiving - The contents of the artefact folder is compressed
+
+* Added uploading to S3
+
+* Flattened folder struture, removed timestamping of folders within the archive to make it easier for the Ingestion process to handle

tasks/DIR-3098/Note 2018-11-08T14.27.59.md

@@ -0,0 +1,21 @@
+DIR-3098
+========================
+
+DIR-3098 Extend OBDFCASCRAPE to be able to upload German NCA data
+
+
+
+ OBDFCASCRAPE extended to enable archiving of the data scraped from https://portal.mvp.bafin.de/database/ZahlInstInfo/?locale=en_GB
+ non-repudiation data for each downloaded file stored in the Artefact Store using path */artefacts/DE/BAFIN/YYYYMMDDHHMM/filename-nr.zip* (see https://openbanking.atlassian.net/wiki/spaces/DEV/pages/465634191/DRAFT+NCA+Data+Acquisition+and+Access+Design+Document)
+ Archived data uploaded to the correct S3 store
+ unit tests included with the source code of the agent
+
+
+
+
+
+
+As a user interested in the Cypriot NCA PSD2 data, I would like to data including non-repudiation data which has been retrieved from Cyprus to be uploaded to the artefact Store.
+
+
+

tasks/DIR-3098/commit.md

@@ -0,0 +1,17 @@
+DIR-3098 Extend OBDFCASCRAPE to be able to upload German NCA data
+
+# Summary
+
+* Added archiving - The contents of the artefact folder is compressed
+
+* Exteded archiving to France.
+
+* Added uploading to S3
+
+* Flattened folder structure, removed timestamping of folders within the archive to make it easier for the Ingestion process to handle
+
+* Added an index of the pages scraped to aid ingestion
+
+* Changed filenames to ensure there are no collisions, items are prefixed with either ps_,  em or ci_.
+
+* Implemented Amazon SQS message service so that an announcement can be made when a new file is avilable for ingestion

tasks/DIR-3111/Sweden Notes.md

@@ -0,0 +1,76 @@
+Sweden Notes
+========================
+
+
+PI
+===
+
+```
+How to access the Swedish information on Payment Institutions The link in the NCA Regulatory Permissions.xls (tab 1), contains the URLs to the NCA for Sweden. Information on individual PIs can be accessed by searching for them, as described here.  Main business = Payment Service Companies THEN 
+
+
+1) Category = Payment Institution 
+
+https://www.fi.se/en/our-registers/company-register/?huvudkategori=Betaltj%C3%A4nstf%C3%B6retag&cat=BET&area=#results 
+
+
+2) Category = Registered Payment Firm 
+
+https://www.fi.se/en/our-registers/company-register/?huvudkategori=Betaltj%C3%A4nstf%C3%B6retag&cat=BETREG&area=#results  
+
+
+
+Note - If you select both categories using the CTRL as specified on the page, it performs an AND condition and will give you an incorrect result. Run the categories individually.  List item(s) reviewed by Mitch.  From the Advanced Search box select from the Main business drop down "Payment Service Companies"  The categories will be displayed in the category box - select (highlight) one from this list or leave as the default selection. Then select the [Search] button. The results will be appended to the page.  Note - Selecting values from the categories list creates an AND condition on the values. Not selecting any values will return all values grouped by category heading.
+
+
+```
+
+
+EMONEY
+===
+
+```
+How to access the Swedish information on Electronic Money Institutions
+The link in the NCA Regulatory Permissions.xls (tab 1), contains two URLs. This is because you have two list values to select and you must run the searches individually, else the query performs an "and" on the values and returns an incorrect result.
+
+Main business = Electronic Money Issuers
+THEN
+1) Category = Swedish E-Money Institution
+https://www.fi.se/en/our-registers/company-register/?huvudkategori=Utgivare+av+elektroniska+pengar&cat=EINST&area=#results
+2) Category = Swedish Registered E-Money Issuer
+https://www.fi.se/en/our-registers/company-register/?huvudkategori=Utgivare+av+elektroniska+pengar&cat=REGUTG&area=#results
+
+Note - If you select both categories using the CTRL as specified on the page, it performs an AND condition and will give you an incorrect result. Run the categories individually.
+
+List item(s) to be selected -  reviewed by Mitch. 
+
+From the Advanced Search box select from the Main business drop down "Electronic Money Issuers"  The categories of EMI will be displayed in the category box - only select one item (as shown above), then select the [Search] button.
+The results will be appended to the page.
+
+```
+
+
+
+CREDIT INSTITUTES
+===
+```
+
+MHow to access the Swedish information on Credit Institutions
+The link in the NCA Regulatory Permissions.xls (tab 1), contains the URLs to the NCA for Sweden. Information on individual Credit Institutions can be accessed by searching for them, as described here.
+
+From the Advanced Search box selct from the Main business drop down "Banks" and then “Searching the registers” drop down select the following three values (you'll need to do them individually):
+
+From the "Main Business" drop down select " Bank
+THEN
+1) Category = Banking Companies (Linited Liability Company)
+https://www.fi.se/en/our-registers/company-register/?huvudkategori=Bank&cat=BANK&area=#results
+2) Category = Members-Bank
+https://www.fi.se/en/our-registers/company-register/?huvudkategori=Bank&cat=MBANK&area=#results
+3) Category = Savings Bank
+https://www.fi.se/en/our-registers/company-register/?huvudkategori=Bank&cat=SPAR&area=#results
+
+Note - If you select both categories using the CTRL as specified on the page, it performs an AND condition and will give you an incorrect result. Run the categories individually. If you put no categories (leave as default) then you will get all categories shown sequentially with headings.
+
+List item(s) reviewed by Mitch.
+
+```

tasks/DIR-3111/commit.md

@@ -0,0 +1,11 @@
+DIR-3111 Extend OBDFCASCRAPE to be able to upload Swedish NCA data
+
+# Summary
+
+* Added archiving - The contents of the artefact folder is compressed
+
+* Added an index of the pages scraped to aid ingestion
+
+* Changed filenames to ensure there are no collisions, items are prefixed with either ps_,  em or ci_.
+
+* Implemented Amazon SQS message service so that an announcement can be made when a new file is avilable for ingestion

tasks/DIR-3119/Note 2018-12-04T14.53.30.md

@@ -0,0 +1,32 @@
+Note 2018-12-04T14.53.30
+========================
+
+**Payment Institute**
+
+How to access the Irish information on Payment Institutions
+https://www.bancaditalia.it/compiti/vigilanza/albi-elenchi/index.html?com.dotmarketing.htmlpage.language=1
+
+https://infostat.bancaditalia.it/GIAVAInquiry-public/ng/int-albi/search
+
+Select the following URL ~~https://infostat.bancaditalia.it/giava-inquiry-public/flex/Giava/GIAVAFEInquiry.html#~~ 
+
+When the page opens you will need to select the English Version hyperlink at the top of the page and confirm the popup message box [Si].
+
+The screen will change to the "Intermediary search terms" screen. The screen (form) displayed has three sections:
+
+1) Historical Search - Do not use.
+
+2) Advanced search in the registers at a given time - use this for CIs in Italy (step 2)
+ ---- In Registers / Lists drop down select -  Albo IP ART.114-Septies Tub = Register of PI (Payment Institutions) 
+ ---- The Reference Date - will default to today
+ ---- Country - you can't enter directly into this field -  I have selected Italia, to do this you need to open the search popup, enter the country (you have to enter more than two characters) and select from the window, then click the [Chosen] button to return the value to the form. 
+
+If you do not select you get all countries.
+Need to check, if I dont use this country select I also get foreign registered banks, what is the correct search? 
+
+Register / Lists  = Albo IP ART.114-Septies Tub = Register of PI (Payment Institutions) 
+Reference Date = defaults to today's date
+Country = Italia
+
+3) Search Advanced cross-border operation to a date  (step 6) You also need to run this query.
+This allows you to see the passporting out.

tasks/DIR-3119/Note 2018-12-06T09.36.48.md

@@ -0,0 +1,3 @@
+Note 2018-12-06T09.36.48
+========================
+

tasks/DIR-3124/Note 2018-12-13T13.56.18.md

@@ -0,0 +1,12 @@
+Note 2018-12-13T13.56.18
+========================
+
+```
+
+
+How to access the Czech information on Credit Institutions
+
+This URL https://apl.cnb.cz/apljerrsdad/JERRS.WEB07.INTRO_PAGE?p_lang=en takes you to the register page for the Czech NCA. Select the "Predefined List 
+
+
+```

tasks/DIR-3124/commit.md

@@ -0,0 +1,31 @@
+DIR-3124 Extend OBDFCASCRAPE to download Czech NCA data
+
+# Summary
+
+* Captcha breaker created
+
+* Czech data scraped
+
+* Data uploaded to S3
+
+* Tests created
+
+```
+# CZECH:: Scrape a Payment Service
+# CZECH::Extract Basic Payment Services from Page
+ok 1 Extracted Details from Page
+# CZECH::Extract More advanced details Payment Services from Page
+ok 2 Extracted Details from Page
+# CZECH::Extract More Authorized activities for Payment Services from Page
+ok 3 Extracted Authorized activities from Page
+# CZECH:: Scrape creditServices Authorised activitites
+# CZECH::Extract Authorized activities ( No 7 section, wih Matrix )
+ok 4 Extracted Details from Page
+# CZECH::Extract Authorized activities ( 7 Section no active, no Matrix )
+ok 5 Extracted Details from Page
+# CZECH::Extract Authorized activities ( With 7 Section and Matrix )
+ok 6 Extracted Details from Page
+# CZECH::Extract Authorized activities ( 4 tables including Matrix )
+ok 7 Extracted Details from Page
+
+```

tasks/DIR-3128 EE/Note 2019-03-19T12.01.52.md

@@ -0,0 +1,39 @@
+Note 2019-03-19T12.01.52
+========================
+
+Showing the first value selected … From here rather than scrape this value use the left hand menu commands to filter and select. Under the Payment institutions menu on the left hand side, the following items need to be run:
+
+https://www.fi.ee/en/payment-services/e-money-institutions/e-money-institutions
+1) Estonian e-money institutions -- Yes -- -No records returned - cannot define template as a result
+https://www.fi.ee/en/payment-services/payment-services/e-money-institutions/estonian-e-money-institutions
+2) Estonian e-money institutions with an exemption -- Yes --- No records returned - cannot define template as a result
+https://www.fi.ee/en/payment-services/payment-services/e-money-institutions/estonian-e-money-institutions-exemption
+3) Disributors of E-money -- Yes --- No records returned - cannot define template as a result
+https://www.fi.ee/en/payment-services/payment-services/e-money-institutions/distributors-e-money
+4) Providers of cross-border e-money services -- Yes --- screen shot below (step 2 and step 3)
+https://www.fi.ee/en/payment-services/e-money-institutions/providers-cross-border-e-money-services
+5) Distributors for providers of cross-border e-money services -- Yes --- screen shots (step 4 & step 5)
+https://www.fi.ee/en/distributors-providers-cross-border-e-money-services
+6) Branches of foreign e-money institutions  -- Yes --- No records returned - cannot define template as a result
+https://www.fi.ee/en/payment-services/payment-services/e-money-institutions/branches-foreign-e-money-institutions
+
+The following screen shot is produced on running:
+4) Providers of cross-border e-money services
+
+Note on the following list - The first two entries have " "  displayed on the corporate names. However, when opening the detail of the record, these do not persist. 
+
+
+
+After running the following URL, select open  "Credit institutions" in the left hand menu to reveal it sub-menu, we need to run each of these.
+https://www.fi.ee/en/banking-and-credit
+
+1) Licensed credit institutions in Estonia --- Yes, we need to run
+https://www.fi.ee/en/banking-and-credit/banking-and-credit/credit-institutions/licensed-credit-institutions-estonia
+2) Affiliated branches of foreign credit institutions --- Yes, we need to run
+https://www.fi.ee/en/banking-and-credit/credit-institutions/affiliated-branches-foreign-credit-institutions
+3) Representative offices of foreign credit institutions --- Yes - no records returned can't evaluate
+https://www.fi.ee/en/banking-and-credit/banking-and-credit/credit-institutions/representative-offices-foreign-credit-institutions
+4) Providers of cross-border banking services --- Yes, we need to run
+https://www.fi.ee/en/banking-and-credit/banking-and-credit/credit-institutions/providers-cross-border-banking-services
+
+made a modiication

tasks/DIR-3159/Note 2018-11-19T10.35.14.md

@@ -0,0 +1,75 @@
+Note 2018-11-19T10.35.14
+========================
+
+
+
+## Payment Institute
+
+```
+How to access the Maltese information on Payment Institutions
+The link in the NCA Regulatory Permissions.xls (tab 1), contains the URL (https://www.mfsa.com.mt/pages/licenceholders.aspx) to the NCA for Malta. Information on individual PIs can be accessed by clicking on them in the URL, as described here.
+
+Open the following General register URL - https://www.mfsa.com.mt/pages/licenceholders.aspx
+In the alternative search frop down fields select:
+Field 1=  Financial Institutions
+Field 2 = Financial isntitutions licensed to undertake payment services under the 2nd Schedule to the Financial Institutions Act (Payment Institutions)
+
+The page will refresh with the query results.
+
+Note - this search will only give you the details of the PIs licensed in Malta. - See Step 3 for Companies licensed in Malta that can passport out.
+
+
+```
+
+```
+Identifying those Maltese Pis that can passport out 
+
+Field 1=  Financial Institutions
+Field 2 =  Local Financial Institutions licensed to undertake activities under the 2nd Schedule to the Financial Institutions Act (Payment Institutions) exercising the freedom to provide services outside Malta
+
+Field 1=  Financial Institutions
+Field 2 =  Local Financial Institutions licensed to undertake activities under the 2nd Schedule to the Financial Institutions Act (Payment Institutions) exercising the freedom to establish a branch outside Malta
+
+The following screen dump show point 1 run and also the opening of a detail row to show the passporting information.
+Screen shot 1 - below - I have picked Mistral, this is also shown in Step 3a as the item returned by the query. 
+Screen shot 2 - below - I have picked Entercash as its agent Pluspay is shown in Step 3b.
+
+
+
+
+
+
+```
+
+
+
+
+```
+
+How to access the Maltese information on Electronic Money Institutions
+The link in the NCA Regulatory Permissions.xls (tab 1), contains the URL (https://www.mfsa.com.mt/pages/licenceholders.aspx) to the NCA for Malta. Information on individual PIs can be accessed by clicking on them in the URL, as described here.
+
+Open the following General register URL - https://www.mfsa.com.mt/pages/licenceholders.aspx
+In the alternative search frop down fields select:
+Field 1=  Financial Institutions
+Field 2 = Financial institutions licenced to issue electronic money under the 3rd Schedule to the Financial Institutions Act (Electronic Money Institutions)
+
+
+
+The page will refresh with the query results.
+
+Note - this search will only give you the details of the EMIs licensed in Malta. - See Step 3 for Companies licensed in Malta that can passport out.
+
+```
+
+
+```
+
+
+Field 1 = Credit Institutions
+Field 2 =  Credit Institutions > Freedom of Services and Establishments - Exercise of the freedom to provide services outside Malta
+ - there are 19 insititutions with the ability to provide services outside of Malta.
+Field 3 =  Freedom of Services and Establishments - Exercise of the freedom to set up an establishment outside Malta
+
+
+```

tasks/DIR-3159/commit.md

@@ -0,0 +1,19 @@
+DIR-3159 Extend OBDFCASCRAPE to download Malta NCA data
+
+# Summary
+
+* Added archiving - The contents of the artefact folder is compressed
+
+* Added an index of the pages scraped to aid ingestion
+
+* Changed filenames to ensure there are no collisions, items are prefixed with either ps_,  em or ci_.
+
+* Implemented Amazon SQS message service so that an announcement can be made when a new file is avilable for ingestion
+
+* Added dictioary loader and auto saver to the main Scraper object.
+
+* Tests for the scraper
+
+* Added the correct whois service for Malta
+
+* Implemented auto mated translate

tasks/DIR-3164 Slovakia/commit.md

@@ -0,0 +1,37 @@
+DIR-3164 Extend OBDFCASCRAPE to download Slovakia NCA data
+
+# Summary
+
+* Slovakia data scraped
+
+* Data uploaded to S3
+
+* Tests created
+
+```
+# SLOVAKIA:: Scrape a Payment Service 001
+# SLOVAKIA::Extract Payment Services Basic Details from Page
+ok 1 Extracted Basic Details from Page
+# SLOVAKIA::Extract Payment Services Table from Page
+ok 2 Extracted Table Details from Page
+# SLOVAKIA:: Scrape a Payment Service 002
+# SLOVAKIA::Extract Payment Services Basic Details from Page
+ok 3 Extracted Basic Details from Page
+# SLOVAKIA::Extract Payment Services Table from Page
+ok 4 Extracted Table Details from Page
+# SLOVAKIA:: Scrape a Payment Service 003
+# SLOVAKIA::Extract Payment Services Basic Details from Page
+ok 5 Extracted Basic Details from Page
+# SLOVAKIA::Extract Payment Services Table from Page
+ok 6 Extracted Table Details from Page
+# SLOVAKIA:: Scrape a CI 
+# SLOVAKIA::Complex CI Data
+ok 7 Extracted Table Details from Page
+
+1..7
+# tests 7
+# pass  7
+
+# ok
+
+```

tasks/DIR-3178 LT/commit.md

@@ -0,0 +1,11 @@
+DIR-3178 Extend OBDFCASCRAPE to download Lithuania NCA data
+
+# Summary
+
+* Lithuania data scraped
+
+* Data uploaded to S3
+
+* Tests created
+
+* Docker container updated to include Lithuania runner

tasks/DIR-3182 LV/commit.md

@@ -0,0 +1,11 @@
+DIR-3182 Extend OBDFCASCRAPE to download Latvia NCA data
+
+# Summary
+
+* Latvia data scraped
+
+* Data uploaded to S3
+
+* Tests created
+
+* Docker container updated to include Latvia runner

tasks/DIR-3200 NO/commit.md

@@ -0,0 +1,11 @@
+DIR-3200 Extend OBDFCASCRAPE to download Norway NCA data
+
+* Norway data scraped
+
+* Data uploaded to S3
+
+* Tests created
+
+* Docker container updated to include Norway runner
+
+* Turned off dictionary saving

tasks/DIR-3232/Note 2018-11-21T14.20.16.md

@@ -0,0 +1,27 @@
+Note 2018-11-21T14.20.16
+========================
+
+DIR-3232 DEV2- Build an Amazon Linux bootstrapper
+
+* Scraper now works inside docker container
+
+* Scraper uploads to S3 correctly
+
+* Scraper now set up to use an internal cron job, so starts at predetermined time.
+
+* Now utilising PM2 to ensure the process restart correctly when the Docke image is rebuilt
+
+* If the image is restarted during a scrape process that process will restart
+
+* Dictionary files are now stored in S3
+ 
+
+
+**** 2018-11-28 ****
+
+* If there is poor network connectevitiy then some pages time out. Trying to make the scraper more resillient.
+
+* Jenkins file created
+
+* Makefile created tested
+

tasks/DIR-3232/commit.md

@@ -0,0 +1,21 @@
+DIR-3232 DEV2- Build an Amazon Linux bootstrapper
+
+# Summary
+
+* Scraper now works inside docker container
+
+* Scraper uploads to S3 correctly
+
+* Scraper now set up to use an internal cron job, so starts at predetermined time.
+
+* Now utilising PM2 to ensure the process restart correctly when the Docke image is rebuilt
+
+* If the image is restarted during a scrape process that process will restart
+
+* Dictionary files are now stored in S3
+ 
+* If there is poor network connectevitiy then some pages time out. Trying to make the scraper more resillient.
+
+* Jenkins file created
+
+* Makefile created tested

tasks/DIR-3370/commit 2.md

@@ -0,0 +1,11 @@
+DIR-3370 OBDFCASCRAPE Chore - Updated functionality trickle down
+
+# Summary
+
+* All page changes wrapped with an error handler
+
+* Czech - extractEntityAuthority now adding data via an array as was intended. Tests updated
+
+* Italy - mode index set back to 0
+
+* Added a handler for unhandled Promise rejections

tasks/DIR-3370/commit.md

@@ -0,0 +1,23 @@
+DIR-3370 OBDFCASCRAPE Chore - Updated functionality trickle down
+
+# Summary
+
+* Added uncaughtException capture to CY, DE, DK, FR, IE, IT, MT, NL, SE
+
+* All processes using process.env.NODE_ENV instead of process.env.PROD
+
+* All index files ( paymentServices.json, emoneyServices.json, creditServices.json ) consistently named and streamlined
+
+* Czech : Fixed output of debug log
+
+* Germany : Added parameters from the url of each page to the index
+
+* France : Changed one debug to an info to show that the scraper is running
+
+* Italy : Info logging tidied
+
+* Italt : A page transition was failing inconsitently
+
+* Malta : Now including the id from the url parameters and the metaStep in the index.
+
+* Malta : ID is appended to the end of the file name to prevent collisions. 

tasks/DIR-3376 Lux/Note 2019-01-16T15.29.30.md

@@ -0,0 +1,12 @@
+Note 2019-01-16T15.29.30
+========================
+
+How to access the Greek information on Payment Institutions
+
+This URL https://supervisedentities.apps.cssf.lu/index.html?language=en#AdvancedSearch takes you to Advances Search page where you may run a query on PIs for Luxembourg. In the drop down list selection, you have the values for; payment institutions, electronic money institutions and credit institutions (banks).
+
+Note - The registers are provided in French (default) you can switch this to English or German. I have selected the English variant (en), see url above.
+
+From the screen below perform the following:
+Type = Payment Institutions
+Select the check box "Register of payment institutions authorised in accordance with the Law of 10 November 2009 relating to payment services". The section of this check box will display the section "Payment services carried out" check all of the rows returned, they are the activities of the companies. Now select the [Search] button. 

tasks/DIR-3376 Lux/Note 2019-01-21T14.03.41.md

@@ -0,0 +1,53 @@
+Note 2019-01-21T14.03.41
+========================
+
+
+Field 1=  Financial Institutions
+Field 2 = Financial isntitutions licensed to undertake payment services under the 2nd Schedule to the Financial Institutions Act (Payment Institutions)
+
+
+Field 1=  Financial Institutions
+Field 2 =  Local Financial Institutions licensed to undertake activities under the 2nd Schedule to the Financial Institutions Act (Payment Institutions) exercising the freedom to provide services outside Malta
+
+Field 1=  Financial Institutions
+Field 2 =  Local Financial Institutions licensed to undertake activities under the 2nd Schedule to the Financial Institutions Act (Payment Institutions) exercising the freedom to establish a branch outside Malta
+
+##
+
+
+Field 1=  Financial Institutions
+Field 2 = Financial institutions licenced to issue electronic money under the 3rd Schedule to the Financial Institutions Act (Electronic Money Institutions)
+
+
+Field 1=  Financial Institutions
+Field 2 = Local Financial Institutions licensed to issue electronic money under the 3rd Schedule to the Financial Institutions Act (Electronic Money Institutions) exercising the freedom to establish a branch outside Malta
+
+
+
+Field 1=  Financial Institutions
+Field 3  = Local Financial Institutions licensed to issue electronic money under the 3rd Schedule to the Financial Institutions Act (Electronic Money Institutions) exercising the freedom to provide services outside Malta.
+
+
+
+
+
+
+How to access the Maltese information on Electronic Money Institutions
+The link in the NCA Regulatory Permissions.xls (tab 1), contains the URL (https://www.mfsa.com.mt/pages/licenceholders.aspx) to the NCA for Malta. Information on individual PIs can be accessed by clicking on them in the URL, as described here.
+
+Open the following General register URL - https://www.mfsa.com.mt/pages/licenceholders.aspx
+In the alternative search frop down fields select:
+Field 1=  Financial Institutions
+Field 2 = Financial institutions licenced to issue electronic money under the 3rd Schedule to the Financial Institutions Act (Electronic Money Institutions)
+
+
+
+The page will refresh with the query results.
+
+Note - this search will only give you the details of the EMIs licensed in Malta. - See Step 3 for Companies licensed in Malta that can passport out.
+
+
+Field 1=  Financial Institutions
+Field 2 = Local Financial Institutions licensed to issue electronic money under the 3rd Schedule to the Financial Institutions Act (Electronic Money Institutions) exercising the freedom to establish a branch outside Malta
+Field 3  = Local Financial Institutions licensed to issue electronic money under the 3rd Schedule to the Financial Institutions Act (Electronic Money Institutions) exercising the freedom to provide services outside Malta.
+

tasks/DIR-3376 Lux/comit.md

@@ -0,0 +1,13 @@
+DIR-3376  Extend OBDFCASCRAPE to download Luxembourg NCA data
+
+# Summary
+
+* Luxembourg data scraped
+
+* Data uploaded to S3
+
+* Tests created
+
+* Docker container updated to include Luxembourg runner
+
+* Additional: small update to Malta to extend pages scraped

tasks/DIR-3380 Poland/commit.md

@@ -0,0 +1,13 @@
+DIR-3380  Extend OBDFCASCRAPE to download Poland NCA data
+
+# Summary
+
+* Poland data scraped
+
+* Data uploaded to S3
+
+* Tests created
+
+* Docker container updated to include Poland runner
+
+* Node includes updated, unused items discarded

tasks/DIR-3380 Poland/email.md

@@ -0,0 +1,31 @@
+email
+========================
+
+Having reviewed the missing data:
+
+
+Some of the firm details are missing in the dataset as follows
+
+Local services provided in Poland
+
+1. Co-op Banks  (Missing in dataset)
+2. Commerce Banks (Art 70) (found in ci_creditServices_article70.json)
+3. Joint Stock Companies  (Missing in dataset)
+4. Register of Loan Institution (xls) (Missing in dataset)
+
+
+Cross Border Activities provided in EEA
+ - found in creditServices.json and associated firm files
+
+Can you confirm whether  you would be covering these missing items ?
+
+
+
+
+The data I made available at that point was an incomplete subset of the data to allow you to get started with the importer.
+
+Co-op banks data and Joint Sock companies, these data sets appear to be just contact data.
+
+The register of Loan Institution xls downloader was completed after I made the data available.
+
+

tasks/DIR-3384 Portugal/commit.md

@@ -0,0 +1,11 @@
+DIR-3384 Extend OBDFCASCRAPE to download Portugal NCA data
+
+# Summary
+
+* Portugal data scraped
+
+* Data uploaded to S3
+
+* Tests created
+
+* Docker container updated to include Portugal runner

tasks/DIR-3527 Malta V2/commit.md

@@ -0,0 +1,29 @@
+DIR-3527 Rework OBDFCASCRAPE Malta NCA data
+
+# Summary
+
+* Test data updated with latest versions of the html
+
+* Page indexer updated
+
+* Entity processors updated
+
+* Some of the new source data is potentially broken, had to put in measures to handle the broken data.
+
+* Tests passing
+
+
+```
+# MALTA:: Scrape a Credit Instititute
+# Malta::Extract Details from Page 1
+ok 1 Extracted Details from Page
+# Malta::Extract Details from Page 2
+ok 2 Extracted Details from Page
+# Malta::Extract Details from Page 3
+ok 3 Extracted Details from Page
+# Malta::Extract Details with bullet points
+ok 4 Extracted Details with bullet points Page
+# Malta::Extract Details fom a posibly broken page
+ok 5 Extracted Details with bullet points Page
+
+```

tasks/Index.md

@@ -0,0 +1,9 @@
+Task Index
+========================
+
+DIR-3370 OBDFCASCRAPE Chore - Updated functionality trickle down
+
+Euan Wilson <euan.wilson@plexusrs.com>
+Aaron Cox <A.Cox@roc-search.com>
+Johnny Grill <johnny.grill@damiagroup.com>
+Gavin Marriott <gavin.marriott@gcsrecruitment.com>

tasks/NCA Scraper TimeLine.md

@@ -0,0 +1,132 @@
+NCA Scraper TimeLine
+========================
+
+Done
+---
+* Ireland
+* Netherlands
+* France
+* Cyprus
+* Sweden
+* Germany
+* Malta (started, so will finish)
+* Czech Republic (started, so will finish)
+* Italy
+* Luxemburg
+* Slovakia
+* Portugal
+
+* Poland
+
+
+New List
+===
+| ↕️ | Country | ISO | Status | Scrape Duration |
+|--- |---|---|---| --- |
+| 1 | UK | UK | ? |    |
+| 2 | IRELAND | IE | Done | 00:01:46.208 |
+| 3 | Netherlands | NL | Done |    |
+| 4 | France | FR | Done |    |
+| 5 | Cyprus | CY | Done | 00:01:07.651 |
+| 6 | Sweden | SE | Done |    |
+| 7 | Germany | DE | Fixed DIR-3651 |    |
+| 8 | Italy | IT | Done |    |
+| 9 | EBA REgister | | ? |    |
+| 10 | Luxembourg | LU | Done DIR-3376 |    |
+| 11 | Poland | PL | Done DIR-3380 | 24h-ish |
+| 12 | Portugal  | PT | Done DIR-3384 |    |
+| 13 | Slovakia | SK | Done DIR-3164 |    |
+| 14 | *Denmark* | *DK* | *BLOCKED* |    |
+| 15 | *Hungary* | *HU* | *BLOCKED* |    |
+| 16 | Finland | | *Waiting on JA* |    |
+| 17 | Norway | NO | Done DIR-3200 |    |
+| 18 | Lithunia | LT | Done DIR-3178 | 00:26:46.050 |
+| 19 | Latvia | LV | DIR-3182 | 00:12:04.342    |
+| 20 | Estonia | | ? |    |
+| 21 | Gibraltar | | PATRICK |    |
+| 22 | Belgium | | ? |    |
+| 23 | Greece | | ? |    |
+| 24 | Malta | MT | RE-Done DIR-3527 |    |
+| 25 | Czech Republic | CZ | Done |    |
+| 26 | Bulgaria | | ? |    |
+| 27 | Slovenia | | ? |    |
+| 28 | Austria | | ? |    |
+| 29 | Romania | | ? |    |
+| 30 | Croatia  | | ? |    |
+| 31 | Spain  | | ? |    |
+| 32 | Iceland  | | Not yet |  ?  |
+| 33 | Liechtenstein  | | Not yet | ?   |
+| 34 | Isle Of Man  | | Not yet |  ?  |
+| 35 | Jersey  | | Not yet |  ?  |
+| 36 | Guernsey  |  | Not yet |  ?   |
+
+
+⚠️ ℹ️
+
+
+
+
+16th January release (Pre-Production)
+---
+Note: delayed by 2 weeks due to new infrastructure build
+
+PPE R4-FCA Data
+
+NCA data available in Pre-Production
+
+* FCA
+* Ireland
+* Netherlands
+* France
+* Cyprus
+* Sweden
+* FCA Data Viewer
+* FCA passporting out data entry enabled via Data Viewer
+
+29th January NCA releases (Pre-Production) 
+---
+(Need to schedule a release for Sandbox for whats available)
+Prod R5-NCA Data
+
+NCA data will become available in Pre-Production as they are delivered:
+
+* Germany
+* Malta (started, so will finish)
+* Czech Republic (started, so will finish)
+* Italy
+* EBA Register
+* Luxemburg
+* Poland
+* Portugal
+* Slovakia
+
+
+26th February NCA releases (Pre-Production)
+---
+PPE R6-NCA Data
+
+NCA data will become available in Pre-Production as they are delivered:
+
+Denmark
+Estonia
+Finland
+Hungary
+Belgium
+Greece
+NCA ingestion for P19
+
+
+NCA countries that have not transposed:
+
+Spain
+Norway 
+Iceland
+Liechtenstein
+Gibraltar
+
+added extra:
+
+* Bulgaria
+* Austria
+* Sloveina
+* Romania

tasks/Note 2019-01-28T11.14.10.md

@@ -0,0 +1,216 @@
+```
+
+const cheerio = require('cheerio');
+const path = require('path');
+const jsonfile = require('jsonfile');
+const removeAccents = require('remove-accents-diacritics');
+const logger = require('log4js').getLogger('SK');
+const url = require('url');
+
+logger.level = process.env.LOGGER_LEVEL || 'warn';
+
+class SKScrape extends Scraper {
+
+  constructor() {
+    super();
+    this.id = 'SK';
+
+    this.on('done', () => {
+      this._done();
+    });
+
+    if (process.env.NODE_ENV === 'production')
+      this._checkLock().then((l) => {
+        if(l)
+          this.run();
+      });
+  }
+
+  async processNewPage() {
+    // give the page a few seconds to settle
+    await this._randomWait(this.page, 3, 5);
+
+    const pageUrl = url.parse(await this.page.url());
+
+    switch (pageUrl.pathname) {
+
+      case '/en/our-registers/company-register/':
+        await this.indexRedirector();
+        break;
+
+      case '/en/our-registers/company-register/details':
+        await this.processRedirector();
+        break;
+      case '/en/our-registers/company-register/gransoverskridandehandel/':
+        await this.crossBorderRedirector();
+        break;
+
+      default:
+        if (process.env.NODE_ENV) {
+          await this._uploadError();
+          throw new Error(`Unknown page: ${pageUrl}`);
+        }
+        else {
+          logger.warn('processNewPage Fell through');
+          logger.warn('currentPage.location', pageUrl);
+        }
+        break;
+
+    }
+  }
+
+  async start() {
+    super._start();
+    try {
+      this.mode = 0;
+
+      this.paymentServices = {
+        'items': 0,
+        'links': [],
+        'step': 0,
+        'indexStep': 0,
+        'visited': false,
+        'done' : false,
+        'urls': ['https://www.mfsa.com.mt/pages/licenceholders.aspx'],
+        'sections' : [],
+        'sectionLinks' : []
+      };
+
+      this.emoneyServices = {
+        'items': 0,
+        'links': [],
+        'step': 0,
+        'indexStep': 0,
+        'visited': false,
+        'done' : false,
+        'urls': [],
+        'sections' : [],
+        'sectionLinks' : []
+      };
+
+      this.creditServices = {
+        'items': 0,
+        'links': [],
+        'step': 0,
+        'indexStep': 0,
+        'visited': false,
+        'done' : false,
+        'searchDone' : false,
+        'started': false,
+        'urls': [],
+        'sections' : [],
+        'sectionLinks' : []
+      };
+
+      this.startPage = this.paymentServices.urls[0];
+      this.emoneyUrl = this.emoneyServices.urls[0];
+      this.credit = this.creditServices.urls[0];
+
+      this.setPath(path.resolve(`${__dirname }/../artefacts/MT/MFSA`));
+
+      // await this._doNonRepudiation();
+
+      await this._initBrowser(false);
+      this.page = await this.browser.newPage();
+
+      this.page.on('domcontentloaded', () => {
+        this.processNewPage();
+      });
+
+      this.on('entityComplete', () => {
+        this.handleEntityComplete();
+      });
+
+      this.on('serviceDone', async function() {
+        switch (this.mode) {
+
+          case 0:
+            this.emit('paymentServicesDone');
+            break;
+
+          case 1:
+            this.emit('emoneyServicesDone');
+            break;
+
+          case 2:
+            this.emit('creditServicesDone');
+            break;
+
+        }
+      });
+
+      this.on('paymentServicesDone', async function() {
+        logger.warn('paymentServicesDone');
+        try{
+          this.paymentServices.done = true;
+          jsonfile.writeFileSync(`${this.path}/paymentServices.json`, { 'links': this.paymentServices.links });
+          jsonfile.writeFileSync(`${this.debugPath}/paymentServices.json`, this.paymentServices);
+
+          this.mode++;
+          this.inProgress = false;
+
+          await this._goto(this.emoneyServices.urls[0]);
+        }
+        catch (e) {
+          logger.error(e);
+        }
+      });
+
+      this.on('emoneyServicesDone', async function() {
+        logger.warn('emoneyServicesDone');
+        try{
+          this.emoneyServices.done = true;
+          jsonfile.writeFileSync(`${this.path}/emoneyServices.json`, { 'links':this.emoneyServices.links });
+          jsonfile.writeFileSync(`${this.debugPath}/emoneyServices.json`, this.emoneyServices);
+          this.mode++;
+          this.inProgress = false;
+
+          await this._goto(this.creditServices.urls[0]);
+        }
+        catch (e) {
+          logger.error(e);
+        }
+      });
+
+      this.on('creditServicesDone', async function() {
+        logger.warn('creditServicesDone');
+        try{
+          this.creditServices.done = true;
+          jsonfile.writeFileSync(`${this.path}/creditServices.json`, { 'links':this.creditServices.links });
+          jsonfile.writeFileSync(`${this.debugPath}/creditServices.json`, this.creditServices);
+          this.mode++;
+          this.inProgress = false;
+
+          this.emit('done');
+        }
+        catch (e) {
+          logger.error(e);
+        }
+      });
+
+      //
+
+      await this.page.setViewport({ 'width': 1200, 'height': 800 });
+      await this._goto(this.startPage, { 'waitUntil':'networkidle0' });
+
+      await this._randomWait(this.page, 3, 5);
+    }
+    catch(e) {
+      throw new Error(e);
+    }
+  }
+
+  async run() {
+    await this.start();
+  }
+
+}
+
+module.exports = SKScrape;
+
+
+
+
+
+
+```

tasks/Note 2019-01-31T12.25.17.md

@@ -0,0 +1,25 @@
+Note 2019-01-31T12.25.17
+========================
+
+1. Tjänster kontantinsättning på betalkonto
+
+
+2. Tjänster kontantuttag från betalkonto
+
+3. Genomförande av betaltransaktioner
+a) Inlåning
+b) Genomföra betaltransaktioner
+c) Insättning/uttag betalkonto
+
+4. Genomförande av betaltransaktioner gm kreditutrymme
+a) Inlösa transaktionsbelopp
+b) Ge ut betalningsinstrument/lösa in transaktionsbelopp
+c) Genomföra betalningstransaktioner genom kreditutrymme
+
+5. Utfärdande och/eller förvärvande av betalningsinstrument
+
+6. Penningöverföring
+
+7. Betalningsinitieringstjänster
+
+8. Kontoinformationstjänster

tasks/Note 2019-03-19T12.00.59.md

@@ -0,0 +1,3 @@
+Note 2019-03-19T12.00.59
+========================
+

tasks/OB AWS.md

@@ -0,0 +1,42 @@
+OB AWS
+========================
+
+```
+
+Hi Martin
+
+OB AWS account
+
+https://rax-e4bacea2ff104e308293609ef5cbcb9d.signin.aws.amazon.com/console
+
+username: martin.donnelly
+password: tE2MKv\D9A4fRo$K
+
+group: arn:aws:iam::608578871059:group/DevReadOnly​
+
+Please enable MFA when logged in.
+
+Thanks
+
+Seán
+
+
+```
+
+Switch role
+
+```
+account: 482681734622
+
+role: OB_ReadOnly_Role
+
+
+
+RS Development	
+rsdev	
+openbankng.cc 
+openbanking.xyz	
+
+https://rax-b7a6c98d9ab74eecaae6d4d92b5cd3cb.signin.aws.amazon.com/console
+
+```

tasks/Worklog.md

@@ -0,0 +1,31 @@
+Worklog
+========================
+
+
+##### 2018-11-05 // Monday #####
+
+Started DIR-3059 Extend OBDFCASCRAPE to be able to download Cypriot NCA Credit Institute data
+
+##### 2018-11-06 // Tuesday #####
+
+Continuing with DIR-3059
+
+**DONE** DIR-3059
+
+##### 2018-11-06 // Tuesday #####
+
+Started DIR-3089 Extend OBDFCASCRAPE to be able to upload Irish NCA data
+
+
+##### 2018-11-13 // Tuesday #####
+
+Started DIR-3111 Extend OBDFCASCRAPE to download Swedish NCA data
+
+
+
+
+##### 2018-11-19 // Monday #####
+
+**DONE** DIR-3111 Extend OBDFCASCRAPE to download Swedish NCA data
+
+Started DIR-3159 Extend OBDFCASCRAPE to download Malta NCA data