Scheduling the scrapes

• Schedule the scrapes to run in batches (e.g. 20 at a time, once every 6 hours - i.e 80 a day -  to minimise the chance of LinkedIn security blocking the IP
  • I recognise that for  someone like me with 1.5k contacts, the process will take ~20 days, but that is ok. 
    • Perhaps email progress report (and a file with a subset of the users) after each run  with the updates so far  (once we have fully tested this)
• I have created a linked-inexport@atts-systems.com email address (password Descartes99) for the email to be sent from

Scrape

• When a LinkedIn contact has something unusual in their name (eg the maiden name in brackets, or a same or PhD in brackets) then it fails to allocate the first name and last name successfully.  
  • Suggestion:  Take anything in between brackets and delete (including deleting the brackets)
    • See attachment for examples in my Connections list (6 out 1,222 ‘fail’)
• Scraping of gender from linkedIn needs to be trimmed for it to be effective - returns “…….She/Her……”
  • Let's test it

Settings

• The settings for MaxLinkedInContacts and LinkedInScrapeBatch don't seem to be effective.  We should use these fields to reduce the initial scrape to make testing quicker

Popup Message

• The popup message appears each time your launch Index - “There are 1222 connections. The system pulls data in the batch of 20 per hour. It will take approximately 62 hours or 2 days and 14 hours to completely fetch all the data of 1222 connections. Our system will auto pull data in the interval of 1 hour. You can check status in every one hour. To continue please start the process.”

Exports

• For the PartnerFirm export add extra columns to the XLS export
  • Name of PartnerFirm Users who is connected to the LinkedInContact
• Solve the multiple VCF export 

Linkedin icon

• Change gitignore or other solution to make the LinkedIn icon show