Set an scheduler with web scraping task then save data to database

Jenilyn Sanchez asked on September 10, 2018 10:16

Hi guys! I'm looking for a suggestion on what is the easiest and best practice way on how to achieve below scenario:

In my homepage I have a dynamic text content which will come from external website so basically I need to do a web scrapping to get the data.

I was planning to set an scheduler task to call the API for web scrapping (set to execute the task every 5 minutes) and save the data to the database. So in my web part I will just query in the database to get the data every page load.

Thanks! Appreciate your suggestion.

Correct Answer

Brenden Kehren answered on September 10, 2018 20:15

I'd suggest creating a custom module to store the data or if it's pretty simple data, the a custom table. In your scheduled task, do your work to get the data from the scrape, and save it to the custom module class table or custom table. Should be a pretty simple setup either way you choose.

A custom module or custom table used in conjunction with a scheduled task is pretty simple and IMHO a better setup than using the Integration Bus.

1 votesVote for this answer Unmark Correct answer

Recent Answers

Peter Mogilnitski answered on September 10, 2018 19:18

I would try to use integration bus.

1 votesVote for this answer Mark as a Correct answer

Jenilyn Sanchez answered on September 17, 2018 06:42 (last edited on September 17, 2018 14:11)

0 votesVote for this answer Mark as a Correct answer