U4-2701 - The media creation process needs to be optimized, there's far too much IO occuring

Created by Shannon Deminick 23 Aug 2013, 01:45:48 Updated by Shannon Deminick 23 Aug 2013, 03:41:49

During the media creation/upload process, we are reading/writing to disk far too often, this can be highly optimized.

Here's the process as it occurs when doing the bulk media upload for each item (in regards to IO)

  • Add File -> check exists
  • Ensure directory -> creates directory
  • Create file -> copy uploaded stream to file
  • Create default thumbnail size (100px) ** 2 x calls to GetDimensions to get the dimensions of the original image (one call for width, one call for height!) *** Each call to GetDimensions does: Open file, create an image from stream, return width + height ** Call DoResize -> Opens the original file again, and again creates an image from the stream *** Call GenerateThumbnail **** Check file exists **** Ensure directory -> directory already exists **** Create file
  • Create additional thumbnail sizes (repeat above process)
  • Set media items's property values such as file size, image height, etc... ** set width -> Call GetDimensions again (see above for IO process) ** set height -> Call GetDimensions again (see aboove for IO process) ** set file size -> Go re-open the original file (again) and then get it's file size

I think it's pretty easy to gather that we can make this process about a zillion times faster.

Comments

Shannon Deminick 23 Aug 2013, 03:40:41

I've optimized this and have reduced the CPU overhead by nearly 60% and that is just CPU, I've reduced the number of IO operations significantly. If you weren't running on SSD then this change would make this seems much quicker and then there's the CPU improvements too.

Based on profiling the CPU cycles here's the result for the uploading operation tree: http://screencast.com/t/7bnlRcgRCeyQ


Shannon Deminick 23 Aug 2013, 03:41:34

Fixed in ec4f6de6317805d60903c877b95a5b4e8057027f


Priority: Normal

Type: Bug

State: Fixed

Assignee: Shannon Deminick

Difficulty: Normal

Category:

Backwards Compatible: True

Fix Submitted:

Affected versions: 4.10.0, 4.11.0, 6.0.0, 4.11.4, 6.0.1, 4.11.5, 6.0.2, 4.11.6, 6.0.3, 6.0.4, 4.11.7, 6.1.1, 6.0.6, 4.11.9, 6.0.5, 4.11.8, 6.0.7, 6.1.2, 4.10.1, 4.11.10, 6.1.3, 6.1.4

Due in version: 6.2.0

Sprint:

Story Points:

Cycle: