1

I am trying to compress large datasets (from 15-400 GB each). I am looking for the best compression software for Mac that has

  1. High compression rate.
  2. Is fast

I have tried 7zX, which has excellent compression rate on Ultra settings, but is very slow. I wonder if I can change the settings so it

  1. uses all cores I have
  2. the max RAM
  3. and max CPU.

So I can shorten that time. Or if there is another software that offers this option. Of, course if anybody can recommend any better software, I'd be happy to here from it.

Also about workflows on how to deal with big files that need to be compressed/decompressed later.

Thanks.

fixer1234
  • 27,064
  • 61
  • 75
  • 116
digit
  • 255
  • 4
  • 11
  • Some algorithms are notoriously more space or time efficient than others, but there is no one best compression software. At the very least, you'll need to specify what sort of data we're talking about, preferably by posting a sample. – Marcks Thomas Jun 19 '12 at 10:11
  • mostly video, images and data with a lot (!) of small files – digit Jun 19 '12 at 10:18
  • Compressed video or uncompressed, compressed images or uncompressed, what kind of data, text files or binary? If the videos and images are already compressed (e.g. MPEG 4 and JPEG) I would go for the fastest compression (pigz would be a candidate), since you will not gain much. – Marco Jun 19 '12 at 10:29
  • It seems like you want to use 7z and just wait for it to finish. Of course you do understand any sort of compression means you have to uncompress the files also right? In other words at such a small size you might just want to store the data raw. Although the compression can give you one thing you need data verification that the archive isn't corrupt. – Ramhound Jun 19 '12 at 10:57
  • uncompressed everything. – digit Jun 19 '12 at 10:58
  • Yes i do understand that I need to decompress, therefore I would like to find some software that does this quickly (both ways). 7zX is great but I don't want to wait a day to compress and uncompress 400 GB. – digit Jun 19 '12 at 12:28
  • @usersuper - The solution is to get adjust the compression ratio so it takes a reasonable amount of time. The higher the compression ratio the longer its going to take. If it is taking a day to compress 400GB you have it set way to high. The solution of course is get a faster computer if you really do want HIGH compression rates. – Ramhound Jun 19 '12 at 12:45
  • :) thanks. But I have a fast computer, I want to set compression very high and I know that higher compression equals longer waiting periods. I do not want to buy a new computer, but would like to maximize the processing power of the current one. Now if there would be a solution that respects all this, than I would be glad to hear that. – digit Jun 19 '12 at 14:26

2 Answers2

2

There are versions of bzip (pigz) and bzip2 (pbzip2) which are multi-threaded. Both will use 100% of all cores.

Using the maximum RAM doesn't make sense and the algorithms will take as much as necessary to run, not more.

Which algorithm is best, depends on your use case and data, but pigz will probably be one of the fastest and 7z one of the slowest, but with better compression ratio.

Marco
  • 4,306
  • 1
  • 29
  • 31
  • pigz is very fast, but compression is not great (even on comp. level 7) and if I do not specify a file, but a directory it does not compress and I get: "is a directory --skipping". – digit Jun 19 '12 at 11:24
  • The fastest algorithm will probably never be the one with the best ratio. There is no all-in-one algorithm suitable for everything. bzip and pbzip2 (inclusive the multi-threaded variants) are pure compression tools, they don't create archives. You can use tar (`tar cf directory.tar `) to first create an uncompressed archive and then run pigz on the tar archive. – Marco Jun 19 '12 at 11:32
0

You might want to try LRZIP. It uses LZMA compression with a very large window size (up to 900 MB), and thus can reduce redundancies that BZIP-2 or GZIP cannot. It can run multithreaded, as well. Just use the default settings and you are good to go (stay away from the -z option, as it takes forever).

nascente_diskreta
  • 1
  • 2
  • As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Mar 22 '22 at 11:44