Page 1 of 2 1 2 LastLast
Results 1 to 10 of 14
  1. #1
    SHOBiz's Avatar
    Join Date
    Jul 2008
    Location
    Wylie, Texas
    Posts
    3,577

    Question Extreme File Compression Needed

    I currently have 3.12GB of MP3 data that I deliver over the network across the country to some 2,000 locations. I am trying various scenarios to improve this situation. However, one that I have to perform is decreasing the size going over the network. I have tried various compression utilities (and various settings) such as WinRAR, 7-Zip, WinZip, etc... and am only able to reduce the compressed size to 3.02 GB. I know that this data is already compressed so this is not an easy task, but is there something else out there that I could use to reduce the size another 300 -500 MB?
    Thanks!


  2. #2
    Click avatar for tech links/info, donation request K447's Avatar
    Join Date
    Jul 2007
    Location
    near Toronto, Canada
    Posts
    36,569
    +1
    1,272
    Are you streaming the data for real time listening, or making it available for file download?

    Does the result at the receiving end need to be in MP3 format, or could it be delivered in another music file format that offers better compression?

    Are you willing to pay money for a commercial product that does better music compression/delivery?

    Is this a batch file, one time thing, or will you need to compress different music files next time?

    File compression algorithms do not work well with already compressed music data. Compression methods tuned for music files can do better than MP3.

    Is the original music data uncompressed (ie: WAV), or is MP3 files all you have to work with? What MP3 bit-rate are you currently using?

  3. #3
    is the Music the 80's please send it my way

  4. #4
    SHOBiz's Avatar
    Join Date
    Jul 2008
    Location
    Wylie, Texas
    Posts
    3,577
    Quote Originally Posted by K447 View Post
    Are you streaming the data for real time listening, or making it available for file download?

    Does the result at the receiving end need to be in MP3 format, or could it be delivered in another music file format that offers better compression?

    Are you willing to pay money for a commercial product that does better music compression/delivery?

    Is this a batch file, one time thing, or will you need to compress different music files next time?

    File compression algorithms do not work well with already compressed music data. Compression methods tuned for music files can do better than MP3.

    Is the original music data uncompressed (ie: WAV), or is MP3 files all you have to work with? What MP3 bit-rate are you currently using?

    It's MP3 files placed onto the PC and then iTunes is pointed to these files via the iTunes Library in the Users profile. Per Licensing agreements the files cannot be modified in any manner. That is about as far as I can go under NDA...

  5. #5
    axgrider73's Avatar
    Join Date
    Feb 2007
    Location
    Southeast Ohio
    Posts
    2,059
    May I ask why you are looking for compression. Is it to reduce bandwidth usage?? decrease download times??

  6. #6
    SHOBiz's Avatar
    Join Date
    Jul 2008
    Location
    Wylie, Texas
    Posts
    3,577
    Quote Originally Posted by axgrider73 View Post
    May I ask why you are looking for compression. Is it to reduce bandwidth usage?? decrease download times??

    Both, we use SMS and Nomad last time and everything worked fine. Although we had to cram it down the network to get it there in 5 days. We just need to reduce the transport size/time significantly to meet customer (Retail) needs, So far I have tried about 7 different compression tools and Squeez was the only one do better than all of the others with 3.1% reduction in size. I need more like 15-20% to make a difference. ...Buying someones $50k solution is not what we are after either.... I can generally write my own if I have the time.

  7. #7
    Click avatar for tech links/info, donation request K447's Avatar
    Join Date
    Jul 2007
    Location
    near Toronto, Canada
    Posts
    36,569
    +1
    1,272
    Quote Originally Posted by SHOBiz View Post
    ...3.12GB of MP3 data that I deliver over the network across the country to some 2,000 locations
    Quote Originally Posted by SHOBiz View Post
    Both, we use SMS and Nomad last time and everything worked fine. Although we had to cram it down the network to get it there in 5 days.

    We just need to reduce the transport size/time significantly to meet customer (Retail) needs,

    So far I have tried about 7 different compression tools and Squeez was the only one do better than all of the others with 3.1% reduction in size.

    I need more like 15-20% to make a difference...
    Quote Originally Posted by SHOBiz View Post
    It's MP3 files placed onto the PC and then iTunes is pointed to these files via the iTunes Library in the Users profile.

    Per Licensing agreements the files cannot be modified in any manner. That is about as far as I can go under NDA...
    So you are starting with MP3 files, and must deliver those exact same MP3 files at the other end - correct?

    Once data has been compressed by one method (MP3 in this case), applying another compression method on top usually doesn't gain you much.

    Compression works by removing pattern redundancy from within the source files. The better the compression, the less redundancy remains, and the closer the output looks to pseudo random data. And since random data doesn't have much pattern redundancy, there is little opportunity for further compression.

    What I think you need is an algorithm that is tuned to disassemble/unpack the source MP3 music content, then compress it better than MP3 normally does. Then at the delivery end, reverse the process to re-create the original MP3 files. Since you need the output MP3 to be identical to the input MP3, the algorithm must also be lossless.

    There may be such things out there, I just don't know of any.

    ...3GB to 2,000 destinations takes 5 days

    Does that mean each destination takes 5 days to download the file?
    Or just that the aggregate time for all of them to complete is 5 days?

    If it is aggregate time that you need to shrink, have you looked at changing the network distribution model to support something akin to multi-casting, where a single download 'broadcast' can be received by many destinations at once.

    A small number of 'broadcasts' could take care of the bulk of the workload, and one-by-one downloads could handle the remainder.

    A brute force method is to create, or subscribe to, a set of distributed 'download servers', so that you have huge bandwidth available for the targets to pull the files from.

  8. #8
    SHOBiz's Avatar
    Join Date
    Jul 2008
    Location
    Wylie, Texas
    Posts
    3,577
    Quote Originally Posted by K447 View Post
    So you are starting with MP3 files, and must deliver those exact same MP3 files at the other end - correct?

    Once data has been compressed by one method (MP3 in this case), applying another compression method on top usually doesn't gain you much.

    Compression works by removing pattern redundancy from within the source files. The better the compression, the less redundancy remains, and the closer the output looks to pseudo random data. And since random data doesn't have much pattern redundancy, there is little opportunity for further compression.

    What I think you need is an algorithm that is tuned to disassemble/unpack the source MP3 music content, then compress it better than MP3 normally does. Then at the delivery end, reverse the process to re-create the original MP3 files. Since you need the output MP3 to be identical to the input MP3, the algorithm must also be lossless.

    There may be such things out there, I just don't know of any.

    ...3GB to 2,000 destinations takes 5 days

    Does that mean each destination takes 5 days to download the file?
    Or just that the aggregate time for all of them to complete is 5 days?

    If it is aggregate time that you need to shrink, have you looked at changing the network distribution model to support something akin to multi-casting, where a single download 'broadcast' can be received by many destinations at once.

    A small number of 'broadcasts' could take care of the bulk of the workload, and one-by-one downloads could handle the remainder.

    A brute force method is to create, or subscribe to, a set of distributed 'download servers', so that you have huge bandwidth available for the targets to pull the files from.

    Preachin to the choir
    These are the types of projects I get. Here take this already VERY compressed file set and make it half that size. Okay. Okay now make it better. Next I think I like aquisitions and mergers better lol...


    I had multicasting setup and someone that that was a terrible thing (arguments can be made for and against) and is removed now. Unfortunately, setting up distribution servers here would probably take about 5 years of meaningless conference calls first

    If I cannot significantly reduce the transfer size I will likely have to eliminate network transfer and build a large USB replication setup in my lab to replicate 50 or so from a master at one time. Those would then be filled with the content and shipped to the locations. Then the User would insert the USB stick and go to a app install web application. That web application would look on the USB stick for a GUIID and match it to an authorized list and then elevate privledges and install the content (users are not admins). At the next semi annual release they would have to return the usb sticks to get new content and rinse/repeat. Not a fan of the whole counting on the User (manager) with the usb sticks yet. Still in the works...

  9. #9
    SHOBiz's Avatar
    Join Date
    Jul 2008
    Location
    Wylie, Texas
    Posts
    3,577
    Quote Originally Posted by ZMANN View Post
    is the Music the 80's please send it my way

    You gets nuthin
    most of the content is aimed at teenagers anyway.

  10. #10
    Click avatar for tech links/info, donation request K447's Avatar
    Join Date
    Jul 2007
    Location
    near Toronto, Canada
    Posts
    36,569
    +1
    1,272
    Quote Originally Posted by SHOBiz View Post
    ...build a large USB replication setup in my lab to replicate 50 or so from a master at one time. Those would then be filled with the content and shipped to the locations.
    ...return the usb sticks to get new content and rinse/repeat...
    Why USB sticks?

    There is a whole industry that is set up to bulk copy, label, package and ship CD/DVD media, and do it cheaply.

    With appropriate content encoding and security (which it sounds like you have an good understanding of), the discs can be a one time, one way distribution mechanism. No returns, no media re-use.

    Online authorization (as you described for the USB key scenario), and away you go. Unless the target machines have no DVD drives...

Page 1 of 2 1 2 LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. VtechTuned wich file I need ?!!
    By AlexCrete in forum Sea Doo Spark
    Replies: 16
    Last Post: 07-15-2014, 06:55 PM
  2. RXT 215 - Compression - Need Help
    By Rochinha in forum 4-Tec Performance
    Replies: 6
    Last Post: 08-17-2013, 05:54 PM
  3. Replies: 7
    Last Post: 08-15-2013, 10:49 PM
  4. Low compression / need help
    By eye4speed in forum Polaris How To & FAQs
    Replies: 3
    Last Post: 06-03-2008, 09:42 PM
  5. need xlt 800 compression/psi
    By dhoad in forum Yamaha PWC Performance (2-stroke)
    Replies: 1
    Last Post: 12-15-2006, 01:33 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •