Get files in Web directory and show thumbnails

Hi Guys.

I need to extract the filenames from www.test.com/myfiles ( images or MP4 files )

Populate them ( responsive grid ?? ) and show thumbnails. The TMSWEB_ResposiveGrid example is exactly what I need.

On click play the video or show the image. ( I can do this stage )......

Which part do you need help with?

Getting the directory listing. Formatting to json ( ideally to match the sample app )

So I can move forward with that.

Cheers

Is the listing already in a file or do you need to "scrape" a website URL to get the listing? Either way, one approach to get started is to just retrieve whatever is at a given URL. This is from the TMS WEB Core manual in the section on TWebHTTPRequest. To use it, drop a TWebHTTPRequest component on your form and add this to a button click event (for example).

And finally, there is also the promise/await based approach that permits writing code as if it is
sequential but still, underlying it is asynchronously executed.

procedure TForm1.WebButton1Click(Sender: TObject);
var
  req: TJSXMLHttpRequest;
begin
  WebHttpRequest1.URL := 'data.json';
  try
    req := await(TJSXMLHttpRequest, WebHttpRequest1.Perform());
    showmessage(string(req.response));
  except
  // handle failure to execute request here
  end;
end;

Note: do not forget to mark the method WebButton1Click() in the form declaration as async:

TForm1 = class(TWebForm) 
  [async]  procedure WebButton1Click(Sender: TObject);
end;

Here, string(req.response) should end up holding the contents of your listing, whether it is a directory or a file or whatever. You can put whatever you like in the URL, it doesn't have to be a link to a JSON file. Depending on what the response looks like, converting to JSON may be trivial or may take a bit of work. Do you have an example of what the response might look like?

I post various images ( offers ) and videos ( product demos ) into a specific directory.

So I need a mechanism to list all the files in that directory.

I have control of the directory permissions via the CPanel.

So is there a method through WebCore that will allow me to get the list of files.

Right, so if you use the above component, you should end up with a string that has the directory in it. This can then be parsed to get the filenames or sizes or whatever else is present in the directory and then converted to JSON or whatever else you might need. If you post a copy of what the response looks like, I (or someone else lurking about here) can help with converting that into JSON.

Part of the trouble is that different web servers and web server configurations format their output differently, so what you get back from TWebHTTPRequest needs to be reviewed carefully. Here's an example of how it might work. The result of clicking on the button in this example will be a TStringList containing all the filenames. Converting that into JSON is then not very difficult at all.

procedure TForm1.WebButton1Click(Sender: TObject);
var
  req: TJSXMLHttpRequest;
  search: String;
  filename: String;
  filelist: TStringList;
begin
  WebHttpRequest1.URL := 'http://www.sfu.ca/images/';
  try
    req := await(TJSXMLHttpRequest, WebHttpRequest1.Perform());
    search := string(req.Response);
    filelist := TStringList.Create;

    While pos('<a href=', search) > 0 do
    begin
      search := copy(search, pos('<a href="', search)+9, length(search));
      filename := copy(search,1,pos('"',search)-1);
      if pos('jpg',lowercase(filename)) > 0 then
      begin
        filelist.add(filename);
        console.log(filename);
      end;
    end;

  except
  // handle failure to execute request here
  end;
end;

If you point this at another URL, it might bring up different information as it is just parsing out the <a> links which, at the example URL of http://www.sfu.ca/images/, only contains the filenames in the directory. Another URL might have different headers or footers with their own <a> links or perhaps the files themselves might be presented differently than with an <a> tag.

So seeing an example of what you're working with would be a big help.

Hi thanks for the help.

The URL is http:www.toolfolks.com/docs/

I have edited the .htacces file to:

Options +Indexes

Which allows the directory to be listed.

However I get a cross domain error.

I added this to the .htaccess file

Header add Access-Control-Allow-Origin "*"
Header add Access-Control-Allow-Headers "origin, x-requested-with, content-type"
Header add Access-Control-Allow-Methods "PUT, GET, POST, DELETE, OPTIONS"

But still get the cross domain error.

What else do I need to allow the cross domain please

Not sure. Maybe try "Header set" instead of "Header add" but I'm just guessing.

I am still getting the error:
I have the following in the .htaccess file
Header set Access-Control-Allow-Origin ""
Header set Access-Control-Allow-Origin=http://localhost:8000
Options +Indexes
I have added this to the Headers String list ( in the WebHttpRequest component )
Content-Type= text/plain
Accept=
.*
Access to XMLHttpRequest at 'Index of /docs/' from origin 'http://localhost:8000' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.
Any ideas ????

https://www.w3.org/wiki/CORS_Enabled

Open the developer tools in your browser. Go to the network tab and repeat what you do to get the error. This will give you more information.

As far as I can see I have the correct settings in the .htaccess file and in the headers list.

I am getting this in the network tab:

Request URL: http://www.toolfolks.com/docs/

Request Method: GET
Status Code: 200
Referrer Policy: strict-origin-when-cross-origin
Connection: Keep-Alive
content-encoding: gzip
content-type: text/html; charset=UTF-8
date: Tue, 16 Aug 2022 11:55:23 GMT
Keep-Alive: timeout=5, max=100
server: LiteSpeed
transfer-encoding: chunked
vary: Accept-Encoding
x-powered-by: PHP/7.2.34
Accept: .
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Content-Type: text/plain
Host: www.toolfolks.com
Origin: http://localhost:8000
Referer: http://localhost:8000/
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36

When testing here:

I get the directory listing as expected. So I presume there is some other setting I need in the WebHttpRequest component ??????

I created a FMX app and used a IDHttp component using :slight_smile:
memo1.Text := IdHTTP1.Get('Index of /docs/');

I get the listing as expected.

A test with idhttp is no comparison, CORS applies to browser initiated HTTP requests.
Maybe try clearing TWebHttpRequest.Headers

This is not failing. how us something that does!

I have a change of plan. I have used the following PHP file to get the filenames.
If I place the php file in the /docs folder I don't get any Cord errors.

<?php

$dir  = './';
if(is_dir($dir)){
    if($dh = opendir($dir)){
        while(($file = readdir($dh)) != false){
            if($file != "." and $file != ".."){
            
            	if (strpos($file,'.jpg') !== false) {
                $files_array[] = array('Picture' => $file); // Add the file to the array
            } 
            }
        }
    }
    //$return_array =array('name_array' => $files_array);
    $return_array =array( $files_array);
    exit (json_encode($return_array));
}


?>

This outputs the following format

[[{"Picture":"dewaltFlexvoltMitre.jpg"},{"Picture":"DCH253DCD795.jpg"},{"Picture":"chopSaws.jpg"},{"Picture":"54VCutOffSaw.jpg"},{"Picture":"combiKit2amp.jpg"},{"Picture":"combiKits.jpg"},{"Picture":"dewaltSDS.jpg"},{"Picture":"DewaltBrush8.jpg"}]]

How do I map this to get the images into the WebResponsiveGrid

I tried

 WebResponsiveGrid1.Options.ItemTemplate := '<br><img src="http://www.toolfolks.com/docs/(%Picture%)"><br>';
WebResponsiveGrid1.LoadFromJson('http://www.toolfolks.com/docs/files.php');

I had to change the php to

echo (json_encode($files_array));

I now need to manipulate / filter the returned Json data then populate the WebResponsiveGrid.

I have tried the following

procedure TForm1.btnOpenClick(Sender: TObject);
var
 jsonString : String;
 req: TJSXMLHttpRequest;
 js: TJSONObject;
begin
 // WebResponsiveGrid1.Options.ItemTemplate := '<b>(%Brand%) (%Model%)</b><br><i>(%Country%)</i><br><img src="https://download.tmssoftware.com/tmsweb/demos/tmsweb_responsivegrid/img/(%Picture%)"><br>Year:<b>(%Year%)</b><br>Price:<b>(%Price%)€</b><br>';
  WebResponsiveGrid1.Options.ItemTemplate := '<b><center>(%Picture%)</center></b><p style="text-align:center;"><img src="http://www.toolfolks.com/docs/(%Picture%)" height="95%"  width="auto"  ></p><br>';
  WebHttpRequest1.URL := 'https://www.toolfolks.com/docs/files2.php';

    try
    req := await(TJSXMLHttpRequest, WebHttpRequest1.Perform());
    jsonString := string(req.Response);
    js := TJsonObject.ParseJSONValue(jsonString) as TJsonObject;
    WebMemo1.Text := jsonString;
    WebResponsiveGrid1.LoadFromJSON(js);// LoadFromJson(jsonString);

  except
  // handle failure to execute request here
  end;

end;

What is the correct syntax to create a json object from the returned Json then load that into WebResponsiveGrid1

Idiot level please.