2019-06-06

This commit is contained in:
martind2000 2019-06-06 22:03:25 +01:00
parent f0de17cd3b
commit fd91dd1894
14 changed files with 188 additions and 7 deletions

View File

@ -0,0 +1,34 @@
Note 2019-05-30T13.09.58
========================
Evaluate Options to replace Proxy Mesh given procurement review of this supplier:
Option 1: https://oxylabs.io/
Option 2: https://luminati.io/
Conduct a review of each potential supplier and assess best technical and procurement provider
Option 1
---
* Allows Ip whitelisting
* You can use unlimited number of threads.
* It is possible to change the IP on every request.
Prices
| |Residential | | |
|--- |---|---|---| --- |
| Basic | 50 GB included | $12 per GB | $600 min. monthly commitment |
| Pro | 100 GB included | $9 per GB | $900 min. monthly commitment|
Option 2
---

View File

@ -1,4 +1,4 @@
Note 2019-05-29T11.34.48
DIN-329
========================
As discussed, I've added all the defects linked to the Netherlands scrapper to this ticket:
@ -25,11 +25,27 @@ Defect 1
---
The 'CS' ABN Amro Groenbank B V is indexed but not processed
```
[
[
"Statutory name",
"ABN AMRO Groenbank B.V."
],
[
"Trade name",
"ABN AMRO Groenbank B.V."
],
```
Defect 2
---
NIBC Bank N.V. has a category with 2 items but one item is logged.
[
"Category",
"Emittent effecten CSDB, Bank"
]
@ -41,4 +57,6 @@ Coliding filenames
The ID query string value is taken from the href link, and the first 8 characters are extracted as a short hash.
This short hash is added to the filename when it is created.
This short hash is added to the filename when it is created.
Statutory name checking is now implemented. If the name has ben used before, it will use the statuory name and the trade name, otherwise just the statutory name.

View File

@ -0,0 +1,45 @@
DIN-329 - Netherlands NCA Data Defects
# Summary
*Defect 1:*
The menu was using the wrong counter, which gets used later on for processing credit institutes. A new counter was added specifically for building the index.
```
[
[
"Statutory name",
"ABN AMRO Groenbank B.V."
],
[
"Trade name",
"ABN AMRO Groenbank B.V."
],
```
*Defect 2:*
Test created and returns the correct value. It could be possible that the page was updated between scraping and testing.
*Defect 3:*
The ID query string value is taken from the href link, and the first 8 characters are extracted as a short hash.
This short hash is added to the filename when it is created.
Statutory name checking is now implemented. If the name has ben used before, it will use the statuory name and the trade name, otherwise just the statutory name.
# Test Plan
```
# NL:: Scrape a Payment Instititute
...
# NL:: DIN-329 defects
# NL::Defect 2
ok 25 Extracted Details from Page correctly
```

View File

@ -0,0 +1 @@
,martin,Cyan-Corvus-414,05.06.2019 15:46,file:///home/martin/.config/libreoffice/4;

View File

@ -0,0 +1,48 @@
Note 2019-05-30T15.24.02
========================
Spain,
Estonia
Norway
Latvia
Denmark
"IE_CRON",
"NL_CRON",
"FR_CRON",
"CY_CRON",
"SE_CRON",
"DE_CRON",
"IT_CRON",
"LU_CRON",
"LT_CRON",
"PL_CRON",
"SK_CRON",
"PT_CRON",
Ireland, Netherlands, France, Cyprus, Sweden, Germany, Italy, Luxumbourg, Lithuania, Poland, Slovakia, Portugal
Spain, Estonia, Norway, Latvia & Denmark
Deploy Estonia to PPE and test
Activate the entry in the terraform settings, and ensure that the service works correctly with the other services.
https://openbanking.atlassian.net/browse/DIN-342
https://openbanking.atlassian.net/browse/DIN-343
https://openbanking.atlassian.net/browse/DIN-344
https://openbanking.atlassian.net/browse/DIN-345
https://openbanking.atlassian.net/browse/DIN-346

View File

@ -0,0 +1,6 @@
Note 2019-06-03T09.59.54
========================
"NL_CRON",
"IT_CRON",
"PT_CRON",

Binary file not shown.

Binary file not shown.

View File

@ -0,0 +1,13 @@
Note 2019-06-04T12.21.11
========================
Belgium
Czech
Gibralter
Malta
Croatia
Get Belgium, Czech, Gibralter, Malta & Croatia running on PPE.

View File

@ -0,0 +1,19 @@
Note 2019-06-04T12.22.24
========================
Bulgaria
Romania
Austria
Slovenia
Greece
Get Bulgaria, Romania, Austria, Slovenia & Greece running on PPE.
Deploy Greece obdfcascrape to PPE and test
There are 5 countries which need to be deploy to PPE.
Activate the entry in the terraform settings, and ensure that the service works correctly with the other services.

View File

@ -0,0 +1 @@
,martin,Cyan-Corvus-414,04.06.2019 16:27,file:///home/martin/.config/libreoffice/4;

Binary file not shown.

View File

@ -1,4 +1,3 @@
Comment
Note 2019-05-29T16.34.38
========================

View File

@ -1,3 +0,0 @@
Note 2019-05-07T09.47.02
========================