Scala XML crawler, give it a JDBC connection and XML ApiKeys, have the DB filled with XML data.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Az4reus e5445535d9 Readme driven development ahoy! 4 years ago
project Initial check-in 4 years ago
src/main/scala Initial check-in 4 years ago
.gitignore Initial check-in 4 years ago
.scalafmt.conf Initial check-in 4 years ago
README.md Readme driven development ahoy! 4 years ago
build.sbt Initial check-in 4 years ago

README.md

Note: This is not even close to done.

XML Crawler

For when you have more XML keys than you should and also don't have ESI tokens, like the savage you are.

Usage

Into a folder ~/xml-crawler go two files: crawler.conf, and keys.csv. The config is where you specify the JDBC url and config and also lets you tweak various things like the frequency of various things being grabbed, keys is CSV file with the simple format of keyID,vCode.

Effect

By default, upon XML cache timeout, a new request will be fired, and stored together with a timestamp to trace. Please do note that you should probably add a delay to the more spammy requests, else, with any decent number of keys, this will absolutely barrage the API. Logging is implemented and lets you know when the number of backlog requests grows larger than what the XML API lets me do from a single IP, so you can split the keys and run two boxes, should it be necessary.