I'm migrating a site to SS and I've used the CSV bulk uploader to get 6,000+ dataobjects loaded and that all works fine with no significant performance hit. The problem though is that each of those objects has 2-6 images associated with them. I've copied all the images to the assets folder and that automatically synced up fine and loaded all of the 16,000 File objects into the database but I cannot access the /assets section of admin and can not load any of the "select from file store" image selectors without hitting a PHP memory limit. I've bumped the memory limit in PHP up to 1gb and still no luck. I understand 16,000 files is a lot. But it shouldn't be impossible to implement this efficiently. Is there any workaround or known reason for this bottleneck?
As a note, the files are broken up into subdirectories something like this...
assets/CarParts/1/image_1.jpg
assets/CarParts/1/image_2.jpg
assets/CarParts/2/image_1.jpg
assets/CarParts/2/image_2.jpg
There are about 5,000 sub-directories inside CarParts, each containing a handful of images. Is it possible that having a certain ratio of sub-directories to files or a deeper nesting might at least let the asset manager load?