CPC G06F 16/951 (2019.01) [G06F 9/4881 (2013.01); G06F 9/547 (2013.01); G06F 16/953 (2019.01); G06Q 10/105 (2013.01); H04L 63/083 (2013.01); H04L 67/133 (2022.05)] | 20 Claims |
1. A computer-implemented method for determining which servers are available to process web scraping jobs, comprising:
obtaining, by a request intake manager, a first token indicating an identity of a client associated with credentials received with a first application programming interface (API) request from a client computing device, wherein the first API request specifies a target website to scrape and the first token is generated by an authentication service when the credentials are determined to be valid;
selecting, by the request intake manager, a database server to store, in a job database that stores statuses of web scraping jobs while the web scraping jobs are being executed, a job description generated based on the first API request from a plurality of database servers identified as enabled in a database server status table, wherein the plurality of database servers operate independently of one another and each database server is configured to manage data storage of at least a portion of the job database;
assigning, by the request intake manager, a job identifier to the job description, wherein the job identifier encodes an identifier for the selected database server;
transmitting, by the request intake manager, the job identifier to the client computing device; and
transmitting, by the request intake manager, the job identifier and the job description to the selected database server.
|