1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Slow performance opening a file in folder with large number of files

Discussion in 'Windows XP' started by Andrew Meador, Jun 28, 2010.

  1. Andrew Meador

    Andrew Meador Flightless Bird

    I have an Access databse that references image files in a folder.
    There are currently 2700 files in this folder and will reach a total
    of appx 10000 files when data collection is done. The one of the
    database screens allows the user to double-click the image which then
    executes an external program with the path and name of the file to
    open. So the external app gets the exact path and file name to open,
    but the open process takes about 20 seconds to occure. I can go strait
    to the folder where the files are a double click the file and the
    application still takes and equally long time to open the file.
    However, if in a folder with only a small number of files, the app
    opens the image almost instantly. So there appears to be a preformance
    issue with opening even an explicit file in a folder with a large
    number of files. I know I could split the files over multiple folder
    to improve performance, but this require a fair bit of re-coding of
    the database and re-trainined of the users, etc... Is there any
    performance options beides the multiple folder idea? Why does the
    number of files in the folder hind the opening process so badly when
    specific file is targeted?

    Thanks in advance!
    Andrew
     
  2. Daave

    Daave Flightless Bird

    Andrew Meador wrote:
    > I have an Access databse that references image files in a folder.
    > There are currently 2700 files in this folder and will reach a total
    > of appx 10000 files when data collection is done. The one of the
    > database screens allows the user to double-click the image which then
    > executes an external program with the path and name of the file to
    > open. So the external app gets the exact path and file name to open,
    > but the open process takes about 20 seconds to occure. I can go strait
    > to the folder where the files are a double click the file and the
    > application still takes and equally long time to open the file.
    > However, if in a folder with only a small number of files, the app
    > opens the image almost instantly. So there appears to be a preformance
    > issue with opening even an explicit file in a folder with a large
    > number of files. I know I could split the files over multiple folder
    > to improve performance, but this require a fair bit of re-coding of
    > the database and re-trainined of the users, etc... Is there any
    > performance options beides the multiple folder idea? Why does the
    > number of files in the folder hind the opening process so badly when
    > specific file is targeted?
    >
    > Thanks in advance!
    > Andrew


    If you are relying too heavily on the pagefile, adding more RAM will
    yield a tremendous performance benefit.

    How much RAM do you have?
     
  3. Andrew Meador

    Andrew Meador Flightless Bird

    Re: Slow performance opening a file in folder with large number offiles

    On Jun 28, 1:27 pm, "Daave" <da...@example.com> wrote:
    > Andrew Meador wrote:
    > > I have an Access databse that references image files in a folder.
    > > There are currently 2700 files in this folder and will reach a total
    > > of appx 10000 files when data collection is done. The one of the
    > > database screens allows the user to double-click the image which then
    > > executes an external program with the path and name of the file to
    > > open. So the external app gets the exact path and file name to open,
    > > but the open process takes about 20 seconds to occure. I can go strait
    > > to the folder where the files are a double click the file and the
    > > application still takes and equally long time to open the file.
    > > However, if in a folder with only a small number of files, the app
    > > opens the image almost instantly. So there appears to be a preformance
    > > issue with opening even an explicit file in a folder with a large
    > > number of files. I know I could split the files over multiple folder
    > > to improve performance, but this require a fair bit of re-coding of
    > > the database and re-trainined of the users, etc... Is there any
    > > performance options beides the multiple folder idea? Why does the
    > > number of files in the folder hind the opening process so badly when
    > > specific file is targeted?

    >
    > > Thanks in advance!
    > > Andrew

    >
    > If you are relying too heavily on the pagefile, adding more RAM will
    > yield a tremendous performance benefit.
    >
    > How much RAM do you have?- Hide quoted text -
    >
    > - Show quoted text -


    I think I'm ok there. The system has 2GB RAM with only apx 800MB Total
    Commit Charge (on one test system). My dev computer where I have been
    doing most of my dev work is also doing the same thing and it has 3GB
    or RAM with 1.5GB Total Commit Charge
     
  4. John John - MVP

    John John - MVP Flightless Bird

    Re: Slow performance opening a file in folder with large number offiles

    Andrew Meador wrote:
    > On Jun 28, 1:27 pm, "Daave" <da...@example.com> wrote:
    >> Andrew Meador wrote:
    >>> I have an Access databse that references image files in a folder.
    >>> There are currently 2700 files in this folder and will reach a total
    >>> of appx 10000 files when data collection is done. The one of the
    >>> database screens allows the user to double-click the image which then
    >>> executes an external program with the path and name of the file to
    >>> open. So the external app gets the exact path and file name to open,
    >>> but the open process takes about 20 seconds to occure. I can go strait
    >>> to the folder where the files are a double click the file and the
    >>> application still takes and equally long time to open the file.
    >>> However, if in a folder with only a small number of files, the app
    >>> opens the image almost instantly. So there appears to be a preformance
    >>> issue with opening even an explicit file in a folder with a large
    >>> number of files. I know I could split the files over multiple folder
    >>> to improve performance, but this require a fair bit of re-coding of
    >>> the database and re-trainined of the users, etc... Is there any
    >>> performance options beides the multiple folder idea? Why does the
    >>> number of files in the folder hind the opening process so badly when
    >>> specific file is targeted?
    >>> Thanks in advance!
    >>> Andrew

    >> If you are relying too heavily on the pagefile, adding more RAM will
    >> yield a tremendous performance benefit.
    >>
    >> How much RAM do you have?- Hide quoted text -
    >>
    >> - Show quoted text -

    >
    > I think I'm ok there. The system has 2GB RAM with only apx 800MB Total
    > Commit Charge (on one test system). My dev computer where I have been
    > doing most of my dev work is also doing the same thing and it has 3GB
    > or RAM with 1.5GB Total Commit Charge


    The problem seems to be with the application rather than the operating
    system, what do the vendors of the application have to say about this
    latency when opening files? 2,700 files is not really very many files,
    nor is 10,000 for that matter, there could be a delay when
    opening/reading the folder for the first time, but it should be quick on
    subsequent operations.

    John
     
  5. Andrew Meador

    Andrew Meador Flightless Bird

    Re: Slow performance opening a file in folder with large number offiles

    On Jun 28, 2:50 pm, John John - MVP <audetw...@nbnet.nb.ca> wrote:
    > Andrew Meador wrote:
    > > On Jun 28, 1:27 pm, "Daave" <da...@example.com> wrote:
    > >> Andrew Meador wrote:
    > >>> I have an Access databse that references image files in a folder.
    > >>> There are currently 2700 files in this folder and will reach a total
    > >>> of appx 10000 files when data collection is done. The one of the
    > >>> database screens allows the user to double-click the image which then
    > >>> executes an external program with the path and name of the file to
    > >>> open. So the external app gets the exact path and file name to open,
    > >>> but the open process takes about 20 seconds to occure. I can go strait
    > >>> to the folder where the files are a double click the file and the
    > >>> application still takes and equally long time to open the file.
    > >>> However, if in a folder with only a small number of files, the app
    > >>> opens the image almost instantly. So there appears to be a preformance
    > >>> issue with opening even an explicit file in a folder with a large
    > >>> number of files. I know I could split the files over multiple folder
    > >>> to improve performance, but this require a fair bit of re-coding of
    > >>> the database and re-trainined of the users, etc... Is there any
    > >>> performance options beides the multiple folder idea? Why does the
    > >>> number of files in the folder hind the opening process so badly when
    > >>> specific file is targeted?
    > >>> Thanks in advance!
    > >>> Andrew
    > >> If you are relying too heavily on the pagefile, adding more RAM will
    > >> yield a tremendous performance benefit.

    >
    > >> How much RAM do you have?- Hide quoted text -

    >
    > >> - Show quoted text -

    >
    > > I think I'm ok there. The system has 2GB RAM with only apx 800MB Total
    > > Commit Charge (on one test system). My dev computer where I have been
    > > doing most of my dev work is also doing the same thing and it has 3GB
    > > or RAM with 1.5GB Total Commit Charge

    >
    > The problem seems to be with the application rather than the operating
    > system, what do the vendors of the application have to say about this
    > latency when opening files?  2,700 files is not really very many files,
    > nor is 10,000 for that matter, there could be a delay when
    > opening/reading the folder for the first time, but it should be quick on
    > subsequent operations.
    >
    > John- Hide quoted text -
    >
    > - Show quoted text -


    I'll have to follow up with them to see. I guess it could be that
    maybe they pull a file list or something each time, even if the file
    name is specified.
     
  6. HeyBub

    HeyBub Flightless Bird

    Andrew Meador wrote:
    > I have an Access databse that references image files in a folder.
    > There are currently 2700 files in this folder and will reach a total
    > of appx 10000 files when data collection is done. The one of the
    > database screens allows the user to double-click the image which then
    > executes an external program with the path and name of the file to
    > open. So the external app gets the exact path and file name to open,
    > but the open process takes about 20 seconds to occure. I can go strait
    > to the folder where the files are a double click the file and the
    > application still takes and equally long time to open the file.
    > However, if in a folder with only a small number of files, the app
    > opens the image almost instantly. So there appears to be a preformance
    > issue with opening even an explicit file in a folder with a large
    > number of files. I know I could split the files over multiple folder
    > to improve performance, but this require a fair bit of re-coding of
    > the database and re-trainined of the users, etc... Is there any
    > performance options beides the multiple folder idea? Why does the
    > number of files in the folder hind the opening process so badly when
    > specific file is targeted?
    >
    > Thanks in advance!
    > Andrew


    FAT32 or NTFS?

    I can imagine a delay as the system has to meander through a FAT32 directory
    sequentially.

    You might tickle up Process Explorer to discover who's using the cycles
    during a supposed fetch of the file.
     

Share This Page