Full-Time Web Scraping Architect

• Scalability/Performance: Lead and provide expertise in scraping at scale e-commerce marketplaces.
• Data Source Identification: Identify relevant websites and online sources.
• Web Scraping Design: Develop and implement effective web scraping strategies.
• Data Extraction: Create and maintain web scraping scripts or programs.
• Data Cleansing and Validation: Cleanse and validate the collected data to eliminate errors.
• Monitoring and Maintenance: Continuously monitor and maintain the web scraping processes.
• Compliance and Legal Considerations: Stay up-to-date with legal and ethical considerations related to web scraping.
• Documentation: Maintain detailed documentation of web scraping processes.
• Collaboration: Collaborate with other teams to understand data requirements and deliver insights effectively.
• Security: Implement security measures to ensure the confidentiality and protection of sensitive data.

• Proven experience of 7+ years as a Web Scraping Specialist or similar role.
• Expertise in handling dynamic content, user-agent rotation, bypass CAPTCHAs, rate limits, and utilization of proxy services.
• Knowledge on browser fingerprinting.
• Has leadership experience.
• Proficiency in programming languages such as Python, BeautifulSoup, Scrapy, or Selenium.
• Strong knowledge of HTML, CSS, XPath, and other web technologies.
• Knowledge and experience in best of class data storage and retrieval of large volume of scraped data.
• Understanding of web scraping best practices.
• Attention to detail and the ability to handle and process large volumes of data accurately.
• Familiarity with data cleansing techniques and data validation processes.
• Good communication skills.
• Knowledge of web scraping ethics, legal considerations, and compliance with website terms of service.
• Strong problem-solving skills and the ability to adapt to changing web environments

• Bachelor's degree in Computer Science, Data Science, Information Technology, or related fields.
• Experience with cloud-based solutions and distributed web scraping systems.
• Familiarity with APIs and data extraction from non-public sources.
• Knowledge of machine learning techniques.
• Prior experience in handling large-scale data projects and working with big data frameworks.
• Understanding of various data formats such as JSON, XML, CSV, etc.
• Experience with version control systems like Git.

What is the last date for applying to the job?

Which countries are accepted for this remote job?