Update scrapers which have broken over time

 New sites to scrape

• Create login and password field in User for the following sites
  • https://www.securitiesfinancetimes.com/   (no login/password required… has an RSS feed)
  • https://www.globalcustodian.com/ 
  • https://finadium.com/ 
  • https://www.fnlondon.com/
• Put a 5-6 min gap between each article to avoid access denial

User-specific summary email

• Once the broken scrapers are fixed, the main ‘ask’ is to make it “user-specific” (ie each user can define their own list of stories and categories to include in the summary email). So the following need to be linked to the logged user:
  • Headline categories (eg Macro, CV19, Corp, Cryp, Pol, Ukraine, Clearing, x)
  • What to include/exclude for the email
  • What economic stats are key to them
  • Their own list of favourites and commentaries
• At the moment each of these fields are attached to the news article so we need to create a User-News field relationship to achieve this.  Does that make sense to you?

Other

• Make the Economic Stats scrape time 1am and 8.30am
• Cron does not include FirstFT
• Add a button to test that the FT.com login is working
• Full content button (FT) only shows in first block (unassigned)
• Upon user login, run a check on the user's logins and passwords and determine access accordingly

Favicon

•  Not working, both in bookmark and in tab, but only when on the home page… Odd

Login

• Add ability to register

Formatting

• Login button
• Subscription page to show when not logged in
• Ability to look at each headline
• User Edit page -
  • Show categories
  • Button to delete photo
  • Show photo
  • Hide Status from user
• New stories
  • No like/read buttons

Economic stats

• Highlight and suppress buttons not working
• Button to use the Teletrader defaults
• Button to remove your highlight or take standard
• Or if blank use the default - over-ride

Settings

• Add VCard code (Jeroen)

• Review of security (Dashboard -changed to ROLE_USER)

Read count

• The new count needs to be user specific. 
• So this is the number of unread articles in the past 24 hours by user.   Service by user and source
• The read all button should not affect archives
  • Perhaps in archive you can have unread shown separately. 

Removed the CMS and Settings entities

Why isn't LinkedIn password appearing on the list in a User's profile?

Pricing

The functions of the website are

• LinkedIn scrape
• News:
  • Single place to read articles
    • Summary access only
    • Hover for full article
  • Abilty to mark articles as read, to avoid re-reading
  • Abiliy to select key articles and send summary email 
  • Mark economic stats as favourite - to generate an email
  • See what others are liking - are you missing an important well read article?

• Summary Read-only  
  • Full articles available via a link 
• One-stop Read access
  • However
    •  

Economic Market Statistics

• Chron job.  Refresh every 15mins
  • Button to refresh manually if >10mins 
• Historical view by stat

LinkedIn:

• Popup that checks that the LinkedIn login and password are successful
• The first result should return the number of connections and estimate the time to download them, before proceeding.
  • Advise the user how long it will take to download and that a file will be emailed to them 
• Email csv file - one-step directly after the scrape -  ie save a file in the database at the end of the scrap and email (ie merge the 3 buttons we have)

Memberships

• Create a 30-day free trial period
  • Unable to extend
  • Link the systems login to LinkedIn login and other site logins to avoid gaming
  • Buttons to upgrade

Favicon

• Error message occurs when I try to save the Favicon in live.  
• https://newsscraper.atts-systems.com/company_details/edit/1
  • Error message: Unable to write in the "/var/www/html/newsscrapper/public/administration/attachments/Favicons/" directory

Market Data

• Future T+2 data….  

Subscription page/Memberships

• Check whether a user has a membership upon opening the Subscription page
  • If not, create a New membership, initially a Free membership
  • The new Membership button shouldn't open the form, but just save the details as per the button
    • Include Today's date 
    • The new membership button shows all users to Non-Admin users

CompanyDetails

• The HideOther inputs not coming through into Live

LinkedIN scrape (https://www.linkedin.com/in/stephen-j-nurse/)

• Include the summary experience and the employment history

Bugs

• Market Economic Stats scraper doesn't work in live.  Not just United States (filters).  Aman is re-writing the scrape 
• Chron job for  MarketStats doesn't work 

Bugs

• subscriptions_buttons line 45
  • Make this dynamic?
• source\index line 43
  • Make this dynamic?
• Favicon issue on live (when favcicon file exists)

Bugs

• Merge the ‘Highlight/Standard/Low’ buttons into a pop-up.  To avoid confusion. 
• Set User bug in Market Stats

User

Add time zones

LinkedIn contact export

• Develop a view of the contacts that can be expanded/contracted [like Excel]
• Import photo and save file
• Export file - two types: Full CSV and Outlook
  • Export to directory per user
  • Develop a concatenated Notes for the Outlook export
• Control the number of exports (2 now)
• Don't show the flashing screen.  Warn on timings
• Automatically email file

LinkedIn Contacts

• Deleting LinkedIn Contacts -just the owner's contact and the languages spoken

Users

• Delete doesn't work
• Include in user view.
  • Membership

LinkedIn Contacts

• Don't save a Language Spoken for the null case

Linkedin contacts 

• Create a User-Settings upon scrape

User Passwords

• List the users in alphabetical order
• When you edit a User Password it changes the User name

Market stats

• Scrape doesn't work in live

Security

• Add role heirachy in security.yaml

User memberships

• Dates