• Welcome to Tux Reports: Where Penguins Fly. We hope you find the topics varied, interesting, and worthy of your time. Please become a member and join in the discussions.

Slow performance opening a file in folder with large number of files

A

Andrew Meador

Flightless Bird
I have an Access databse that references image files in a folder.
There are currently 2700 files in this folder and will reach a total
of appx 10000 files when data collection is done. The one of the
database screens allows the user to double-click the image which then
executes an external program with the path and name of the file to
open. So the external app gets the exact path and file name to open,
but the open process takes about 20 seconds to occure. I can go strait
to the folder where the files are a double click the file and the
application still takes and equally long time to open the file.
However, if in a folder with only a small number of files, the app
opens the image almost instantly. So there appears to be a preformance
issue with opening even an explicit file in a folder with a large
number of files. I know I could split the files over multiple folder
to improve performance, but this require a fair bit of re-coding of
the database and re-trainined of the users, etc... Is there any
performance options beides the multiple folder idea? Why does the
number of files in the folder hind the opening process so badly when
specific file is targeted?

Thanks in advance!
Andrew
 
D

Daave

Flightless Bird
Andrew Meador wrote:
> I have an Access databse that references image files in a folder.
> There are currently 2700 files in this folder and will reach a total
> of appx 10000 files when data collection is done. The one of the
> database screens allows the user to double-click the image which then
> executes an external program with the path and name of the file to
> open. So the external app gets the exact path and file name to open,
> but the open process takes about 20 seconds to occure. I can go strait
> to the folder where the files are a double click the file and the
> application still takes and equally long time to open the file.
> However, if in a folder with only a small number of files, the app
> opens the image almost instantly. So there appears to be a preformance
> issue with opening even an explicit file in a folder with a large
> number of files. I know I could split the files over multiple folder
> to improve performance, but this require a fair bit of re-coding of
> the database and re-trainined of the users, etc... Is there any
> performance options beides the multiple folder idea? Why does the
> number of files in the folder hind the opening process so badly when
> specific file is targeted?
>
> Thanks in advance!
> Andrew


If you are relying too heavily on the pagefile, adding more RAM will
yield a tremendous performance benefit.

How much RAM do you have?
 
A

Andrew Meador

Flightless Bird
Re: Slow performance opening a file in folder with large number offiles

On Jun 28, 1:27 pm, "Daave" <da...@example.com> wrote:
> Andrew Meador wrote:
> > I have an Access databse that references image files in a folder.
> > There are currently 2700 files in this folder and will reach a total
> > of appx 10000 files when data collection is done. The one of the
> > database screens allows the user to double-click the image which then
> > executes an external program with the path and name of the file to
> > open. So the external app gets the exact path and file name to open,
> > but the open process takes about 20 seconds to occure. I can go strait
> > to the folder where the files are a double click the file and the
> > application still takes and equally long time to open the file.
> > However, if in a folder with only a small number of files, the app
> > opens the image almost instantly. So there appears to be a preformance
> > issue with opening even an explicit file in a folder with a large
> > number of files. I know I could split the files over multiple folder
> > to improve performance, but this require a fair bit of re-coding of
> > the database and re-trainined of the users, etc... Is there any
> > performance options beides the multiple folder idea? Why does the
> > number of files in the folder hind the opening process so badly when
> > specific file is targeted?

>
> > Thanks in advance!
> > Andrew

>
> If you are relying too heavily on the pagefile, adding more RAM will
> yield a tremendous performance benefit.
>
> How much RAM do you have?- Hide quoted text -
>
> - Show quoted text -


I think I'm ok there. The system has 2GB RAM with only apx 800MB Total
Commit Charge (on one test system). My dev computer where I have been
doing most of my dev work is also doing the same thing and it has 3GB
or RAM with 1.5GB Total Commit Charge
 
J

John John - MVP

Flightless Bird
Re: Slow performance opening a file in folder with large number offiles

Andrew Meador wrote:
> On Jun 28, 1:27 pm, "Daave" <da...@example.com> wrote:
>> Andrew Meador wrote:
>>> I have an Access databse that references image files in a folder.
>>> There are currently 2700 files in this folder and will reach a total
>>> of appx 10000 files when data collection is done. The one of the
>>> database screens allows the user to double-click the image which then
>>> executes an external program with the path and name of the file to
>>> open. So the external app gets the exact path and file name to open,
>>> but the open process takes about 20 seconds to occure. I can go strait
>>> to the folder where the files are a double click the file and the
>>> application still takes and equally long time to open the file.
>>> However, if in a folder with only a small number of files, the app
>>> opens the image almost instantly. So there appears to be a preformance
>>> issue with opening even an explicit file in a folder with a large
>>> number of files. I know I could split the files over multiple folder
>>> to improve performance, but this require a fair bit of re-coding of
>>> the database and re-trainined of the users, etc... Is there any
>>> performance options beides the multiple folder idea? Why does the
>>> number of files in the folder hind the opening process so badly when
>>> specific file is targeted?
>>> Thanks in advance!
>>> Andrew

>> If you are relying too heavily on the pagefile, adding more RAM will
>> yield a tremendous performance benefit.
>>
>> How much RAM do you have?- Hide quoted text -
>>
>> - Show quoted text -

>
> I think I'm ok there. The system has 2GB RAM with only apx 800MB Total
> Commit Charge (on one test system). My dev computer where I have been
> doing most of my dev work is also doing the same thing and it has 3GB
> or RAM with 1.5GB Total Commit Charge


The problem seems to be with the application rather than the operating
system, what do the vendors of the application have to say about this
latency when opening files? 2,700 files is not really very many files,
nor is 10,000 for that matter, there could be a delay when
opening/reading the folder for the first time, but it should be quick on
subsequent operations.

John
 
A

Andrew Meador

Flightless Bird
Re: Slow performance opening a file in folder with large number offiles

On Jun 28, 2:50 pm, John John - MVP <audetw...@nbnet.nb.ca> wrote:
> Andrew Meador wrote:
> > On Jun 28, 1:27 pm, "Daave" <da...@example.com> wrote:
> >> Andrew Meador wrote:
> >>> I have an Access databse that references image files in a folder.
> >>> There are currently 2700 files in this folder and will reach a total
> >>> of appx 10000 files when data collection is done. The one of the
> >>> database screens allows the user to double-click the image which then
> >>> executes an external program with the path and name of the file to
> >>> open. So the external app gets the exact path and file name to open,
> >>> but the open process takes about 20 seconds to occure. I can go strait
> >>> to the folder where the files are a double click the file and the
> >>> application still takes and equally long time to open the file.
> >>> However, if in a folder with only a small number of files, the app
> >>> opens the image almost instantly. So there appears to be a preformance
> >>> issue with opening even an explicit file in a folder with a large
> >>> number of files. I know I could split the files over multiple folder
> >>> to improve performance, but this require a fair bit of re-coding of
> >>> the database and re-trainined of the users, etc... Is there any
> >>> performance options beides the multiple folder idea? Why does the
> >>> number of files in the folder hind the opening process so badly when
> >>> specific file is targeted?
> >>> Thanks in advance!
> >>> Andrew
> >> If you are relying too heavily on the pagefile, adding more RAM will
> >> yield a tremendous performance benefit.

>
> >> How much RAM do you have?- Hide quoted text -

>
> >> - Show quoted text -

>
> > I think I'm ok there. The system has 2GB RAM with only apx 800MB Total
> > Commit Charge (on one test system). My dev computer where I have been
> > doing most of my dev work is also doing the same thing and it has 3GB
> > or RAM with 1.5GB Total Commit Charge

>
> The problem seems to be with the application rather than the operating
> system, what do the vendors of the application have to say about this
> latency when opening files?  2,700 files is not really very many files,
> nor is 10,000 for that matter, there could be a delay when
> opening/reading the folder for the first time, but it should be quick on
> subsequent operations.
>
> John- Hide quoted text -
>
> - Show quoted text -


I'll have to follow up with them to see. I guess it could be that
maybe they pull a file list or something each time, even if the file
name is specified.
 
H

HeyBub

Flightless Bird
Andrew Meador wrote:
> I have an Access databse that references image files in a folder.
> There are currently 2700 files in this folder and will reach a total
> of appx 10000 files when data collection is done. The one of the
> database screens allows the user to double-click the image which then
> executes an external program with the path and name of the file to
> open. So the external app gets the exact path and file name to open,
> but the open process takes about 20 seconds to occure. I can go strait
> to the folder where the files are a double click the file and the
> application still takes and equally long time to open the file.
> However, if in a folder with only a small number of files, the app
> opens the image almost instantly. So there appears to be a preformance
> issue with opening even an explicit file in a folder with a large
> number of files. I know I could split the files over multiple folder
> to improve performance, but this require a fair bit of re-coding of
> the database and re-trainined of the users, etc... Is there any
> performance options beides the multiple folder idea? Why does the
> number of files in the folder hind the opening process so badly when
> specific file is targeted?
>
> Thanks in advance!
> Andrew


FAT32 or NTFS?

I can imagine a delay as the system has to meander through a FAT32 directory
sequentially.

You might tickle up Process Explorer to discover who's using the cycles
during a supposed fetch of the file.
 
Top