Update scrapers which have broken over time

 New sites to scrape

• Create login and password field in User for the following sites
  • https://www.securitiesfinancetimes.com/   (no login/password required… has an RSS feed)
  • https://www.globalcustodian.com/ 
  • https://finadium.com/ 
  • https://www.fnlondon.com/
• Put a 5-6 min gap between each article to avoid access denial

User-specific summary email

• Once the broken scrapers are fixed, the main ‘ask’ is to make it “user-specific” (ie each user can define their own list of stories and categories to include in the summary email). So the following need to be linked to the logged user:
  • Headline categories (eg Macro, CV19, Corp, Cryp, Pol, Ukraine, Clearing, x)
  • What to include/exclude for the email
  • What economic stats are key to them
  • Their own list of favourites and commentaries
• At the moment each of these fields are attached to the news article so we need to create a User-News field relationship to achieve this.  Does that make sense to you?

Other

• Make the Economic Stats scrape time 1am and 8.30am
• Cron does not include FirstFT
• Add a button to test that the FT.com login is working
• Full content button (FT) only shows in first block (unassigned)
• Upon user login, run a check on the user's logins and passwords and determine access accordingly

News Lists

• Issue is that the buttons trigger a refresh and slow.
• Can one overcome this with a javascrpt flag and a press to do all button at the end

Users

• Centrally scrape the content of the articles.  Check once the login of the user at their login stage, and track if good.  Then show in the title page if the login is confirmed as valid

Favicon

•  Not working, both in bookmark and in tab, but only when on the home page… Odd

Login

• Add ability to register

Formatting

• Login button
• Subscription page to show when not logged in
• Ability to look at each headline
• User Edit page -
  • Show categories
  • Button to delete photo
  • Show photo
  • Hide Status from user
• New stories
  • No like/read buttons

Economic stats

• Highlight and suppress buttons not working
• Button to use the Teletrader defaults
• Button to remove your highlight or take standard
• Or if blank use the default - over-ride

Settings

• Add VCard code (Jeroen)

• Review of security (Dashboard -changed to ROLE_USER)

Read count

• The new count needs to be user specific. 
• So this is the number of unread articles in the past 24 hours by user.   Service by user and source
• The read all button should not affect archives
  • Perhaps in archive you can have unread shown separately. 

Removed the CMS and Settings entities

Why isn't LinkedIn password appearing on the list in a User's profile?

Pricing

The functions of the website are

• LinkedIn scrape
• News:
  • Single place to read articles
    • Summary access only
    • Hover for full article
  • Abilty to mark articles as read, to avoid re-reading
  • Abiliy to select key articles and send summary email 
  • Mark economic stats as favourite - to generate an email
  • See what others are liking - are you missing an important well read article?

• Summary Read-only  
  • Full articles available via a link 
• One-stop Read access
  • However
    •  

Economic Market Statistics

• Chron job.  Refresh every 15mins
  • Button to refresh manually if >10mins 
• Historical view by stat

LinkedIn:

• Popup that checks that the LinkedIn login and password are successful
• The first result should return the number of connections and estimate the time to download them, before proceeding.
  • Advise the user how long it will take to download and that a file will be emailed to them 
• Email csv file - one-step directly after the scrape -  ie save a file in the database at the end of the scrap and email (ie merge the 3 buttons we have)

Memberships

• Create a 30-day free trial period
  • Unable to extend
  • Link the systems login to LinkedIn login and other site logins to avoid gaming
  • Buttons to upgrade

Favicon

• Error message occurs when I try to save the Favicon in live.  
• https://newsscraper.atts-systems.com/company_details/edit/1
  • Error message: Unable to write in the "/var/www/html/newsscrapper/public/administration/attachments/Favicons/" directory

Market Data

• Future T+2 data….  

Subscription page/Memberships

• Check whether a user has a membership upon opening the Subscription page
  • If not, create a New membership, initially a Free membership
  • The new Membership button shouldn't open the form, but just save the details as per the button
    • Include Today's date 
    • The new membership button shows all users to Non-Admin users

Scrape content

• Check that the scraping of all the news sites works
• Also ensure that the content and not just the headline are scraped saved in the ‘fullContent’ field in the news entity.
• In the view, hovering over the title will display the full content
• For websites where the content is behind a firewall, then login and scrape where available.

Set up a Payment gateway- Stripe? 

CompanyDetails

• The HideOther inputs not coming through into Live

LinkedIN scrape (https://www.linkedin.com/in/stephen-j-nurse/)

• Include the summary experience and the employment history

Bugs

• Market Economic Stats scraper doesn't work in live.  Not just United States (filters).  Aman is re-writing the scrape 
• Chron job for  MarketStats doesn't work 

Bugs

• subscriptions_buttons line 45
  • Make this dynamic?
• source\index line 43
  • Make this dynamic?
• Favicon issue on live (when favcicon file exists)

Bugs

• Merge the ‘Highlight/Standard/Low’ buttons into a pop-up.  To avoid confusion. 
• Set User bug in Market Stats

User

Add time zones

LinkedIn contact export

• Develop a view of the contacts that can be expanded/contracted [like Excel]
• Import photo and save file
• Export file - two types: Full CSV and Outlook
  • Export to directory per user
  • Develop a concatenated Notes for the Outlook export
• Control the number of exports (2 now)
• Don't show the flashing screen.  Warn on timings
• Automatically email file

LinkedIn Contacts

• Deleting LinkedIn Contacts -just the owner's contact and the languages spoken

Users

• Delete doesn't work
• Include in user view.
  • Membership

LinkedIn Contacts

• Don't save a Language Spoken for the null case

Linkedin contacts 

• Create a User-Settings upon scrape

User Password checks

• Need to develop an algorithm to check the passwords upon login

User Passwords

• List the users in alphabetical order
• When you edit a User Password it changes the User name

Deleting LinkedIn Users

• Delete MY Linkedin contacts should not delete them but mark them as hidden
• How to handle when we have a contact shared with multiple "owners"?

Market stats

• Scrape doesn't work in live

Security

• Add role heirachy in security.yaml

User memberships

• Dates

LinkedIn export

• Notes need to include the Employment and Education details
• Searchable text for mission statement
• Export function seems to be failing