I'm trying to make something to let me extract data from a large directory site and the pages are highly structured. But you have to drill down and down to get details, and I only need a few fields from each of the pages that are loaded. It uses a lot of CSS and it's easy to find what I need by simply searching for specific HTML tags in sequence without the need to parse the entire thing, which is quite large.
What's the simplest way to do that in WEB Core? TWebBrowserControl needs a JS function that's operating via an IFRAME, so what I need does not seem simple at all.
Note that I don't really need to SEE the page, so maybe I can load it by passing the URL to an HTTP GET command? If that's the case, then perhaps I could load the page via an edit control then load it into a local browser?
Does the site provide the data using another mechanism, like a JSON file you can download or something like that? What you're likely to run into is CORS restrictions that will typically block your app from retrieving data in the way you describe. It's a bit silly as you can clearly see the data in the browser, but rules are rules and this one trips up a lot of ideas. You can work around CORS sometimes using a server-side tool to retrieve the data - not a browser so CORS doesn't apply, but that's likely the first hurdle to deal with. It could be that there won't be a CORS issue, so you could give the JavaScript fetch command a try and see how you make out.
There's an example of how to test out fetch, and whether CORS will be an issue, here:
Thanks Andrew, I'll try that out. Unfortunately, I replaced the fan on that machine and for some reason when I booted it back up the display is "locked" at 1280 x 1024 and won't go into 4k mode.
This is a public-facing directory site. It might have a REST interface but I don't know why they'd bother.