13

I want to create a large file ~10G filled with zeros and random values. I have tried using:

dd if=/dev/urandom of=10Gfile bs=5G count=10

it creates a file of about 2Gb and exits with a exit status 0. I fail to understand why?

I also tried creating file using:

head -c 10G </dev/urandom >myfile

but it takes about 28-30 mins to create it. But i want it created faster. Anyone has a solution?

Also i wish to create multiple files with same (pseudo) random pattern for comparison. Does anyone know a way to do that? Thanks

Oliver Salzburg
  • 86,445
  • 63
  • 260
  • 306
egeek
  • 251
  • 1
  • 2
  • 4
  • If it's important that the files contain random numbers, that should be part of the title! What means "filled with zeros and random values."? – Volker Siegel Aug 04 '14 at 22:20
  • "I fail to understand why" -- See this question: [Create random data with `dd` and get "partial read warning"...](https://unix.stackexchange.com/q/121865/108618). – Kamil Maciorowski Sep 10 '21 at 13:58

6 Answers6

22

I've seen a pretty neat trick at commandlinefu: use /dev/urandom as a source of randomness (it is a good source), and then using that as a password to an AES stream cipher.

I can't tell you with 100% sure, but I do believe that if you change the parameters (i.e. use way more than just 128 bytes from /dev/urandom), it is at least close enough to a cryptographically secure PRNG, for all practical purposes:

This command generates a pseudo-random data stream using aes-256-ctr with a seed set by /dev/urandom. Redirect to a block device for secure data scrambling.

openssl enc -aes-256-ctr -pass pass:"$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64)" -nosalt < /dev/zero > randomfile.bin

How does this work?

openssl enc -aes-256-ctr will use openssl to encrypt zeroes with AES-256 in CTR mode.

  • What will it encrypt?

/dev/zero

  • What is the password it will use to encrypt it?

dd if=/dev/urandom bs=128 count=1 | base64

That is one block of 128 bytes of /dev/urandom encoded in base64 (the redirect to /dev/null is to ignore errors).

  • I'm actually not sure why -nosalt is being used, since OpenSSL's man page states the following:

      -salt
      use a salt in the key derivation routines. This is the default.

  -nosalt
      don't use a salt in the key derivation routines. This option SHOULD NOT be used except for test purposes or compatibility with ancient versions of OpenSSL and SSLeay.

    Perhaps the point is to make this run as fast as possible, and the use of salts would be unjustified, but I'm not sure whether this would leave any kind of pattern in the ciphertext. The folks at the Cryptography Stack Exchange may be able to give us a more thorough explanation on that.

  • The input is /dev/zero. This is because it really doesn't matter what is being encrypted - the output will be something resembling random data. Zeros are fast to get, and you can get (and encrypt) as much as you want without running out of them.

  • The output is randomfile.bin. It could also be /dev/sdz and you would randomize a full block device.

But I want to create a file with a fixed size! How do I do that?

Simple!

dd if=<(openssl enc -aes-256-ctr -pass pass:"$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64)" -nosalt < /dev/zero) of=filename bs=1M count=100 iflag=fullblock

Just dd that command with a fixed blocksize (which is 1 MB here) and count. The file size will be blocksize * count = 1M * 100 = 100M.

Valmiky Arquissandas
  • 1,855
  • 16
  • 23
  • I was able to generate file quickly but it doesn't stop w/o Ctrl+C.Is there any way of giving out a file size? Also i didn't understand the "-nosalt < /dev/zero >" part.Frm wht i found ol it gives a initialization vector. So in this case is the IV /dev/zero? Also if i want to generate another file with same contents is that possible? – egeek Aug 06 '14 at 18:47
  • It should stop by itself with a "disk full" warning. I'm updating the post to explain how it works. – Valmiky Arquissandas Aug 06 '14 at 18:52
  • I appreciate it.Able to understand clearly now, Thank you! The only issue is generating a output file of a particular size.I need a bunch of them to transfer and check timings n stuff.Can't have it running till the "disk full" warning – egeek Aug 06 '14 at 19:33
  • In that case, you can also use dd. The following line will create a 100 MB file with random data (`count` * `blocksize` = 100 * 1M): `dd if=<(openssl enc -aes-256-ctr -pass pass:"$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64)" -nosalt < /dev/zero) of=filename bs=1M count=100` – Valmiky Arquissandas Aug 06 '14 at 22:36
  • Sorry, the above line doesn't work because `dd` can't handle the input file being a stream. To make it accumulate the input blocks, you need to add the option `iflag=fullblock` to the outer `dd`, like this: `dd if=<(openssl enc -aes-256-ctr -pass pass:"$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64)" -nosalt < /dev/zero) of=filename bs=1M count=100 iflag=fullblock`. I am adding this to the answer. – Valmiky Arquissandas Aug 06 '14 at 22:43
  • `100+0 records in` `100+0 records out` `10695475200 bytes (11 GB) copied, 140.362 s, 76.2 MB/s` `error writing output file` `real 2m22.040s` `user 0m1.065s` `sys 0m23.973s` Although it gives a 10G file in bout 2.5 mins, but it gives a error saying "error in output file", but it does create the file. Not sure why. – egeek Aug 06 '14 at 23:03
  • I was also trying to generate files with the same random pattern.Is that possible?Because rt now i tried generating two files but they differ. (Sorry was trying to post the o/p above as snippet but cudn manage) – egeek Aug 06 '14 at 23:12
  • You can do the `$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64)` step once, and reuse it. I'm pretty sure that will render the same result every time. But why do you want that? – Valmiky Arquissandas Aug 06 '14 at 23:28
  • Yes I think that should work.Wish to generate it at diff locations to check for discrepancies.Thank you @Valmiky Arquissandas! – egeek Aug 06 '14 at 23:37
  • The option -nosalt is nice if you want to create a repeatable file - e.g. if you want to write a pseudorandom content to some device and then verify it was written correctly without having to create a similarily sized file somewhere else. – Dr. Hans-Peter Störr Sep 02 '19 at 09:25
  • @Hans-PeterStörr that was a blast to the past :-) indeed, back then I didn't know why `-nosalt` was there, but it is useful for reproducible outputs, and a salt doesn't offer anything interesting to this use case. I mentioned it could leave a pattern, and that's usually true (probably the reason OpenSSL wrote such a strong warning against its usage), but since the plaintext is `/dev/zero`, I don't think we're caring about that at all. Unless there's an "etiquette" issue, I think I'll update this answer and bring it to 2019. – Valmiky Arquissandas Sep 05 '19 at 17:37
  • Nice, just reporting here that this produces 1.9GB/s on my AMD 5950X. I saw it chewing up a single core to do it. Probably need to use specialized software or start going in parallel to push this faster – Steven Lu Aug 11 '22 at 21:09
14

I'm getting good speeds using the shred utility.

  • 2G with dd in=/dev/urandom - 250sec
  • 2G with openssl rand - 81sec
  • 2G with shred - 39sec

So I expect about 3-4 minutes for 10G with shred.

Create an empty file and shred it by passing the desired file size. 

touch file
shred -n 1 -s 10G file

I'm not sure how cryptographically secure the generated data is, but it looks random. Here's some info on that.

Lyuboslav
  • 320
  • 3
  • 6
5

There is a random number generator program sharand, it writes random bytes to a file. (The program was originally called sharnd, with one letter a less ( see http://mattmahoney.net/dc/)

It takes roughly one third of the time compared to reading /dev/urandom

It's a secure RNG - there are faster, but not secure RNG, but that's not what's needed normally.
To be really fast, look for the collection of RNG algorithms for perl: libstring-random-perl.


Let's give it a try (apt-get install sharand):

$ time sharand a 1000000000                      
sharand a 1000000000  21.72s user 0.34s system 99% cpu 22.087 total

$ time head -c 1000000000 /dev/urandom > urand.out
head -c 1000000000 /dev/urandom > urand.out  0.13s user 61.22s system 99% cpu 1:01.41 total

And the result files - (they do look more random from the inside):

$ ls -l
-rw-rw-r-- 1 siegel siegel 1000000000 Aug  5 03:02 sharand.out
-rw-rw-r-- 1 siegel siegel 1000000000 Aug  5 03:11 urand.out


Comparing the 'total' time values, sharand took only a third of the time needed by the urandom method to create a little less than a GB random bytes:

sharand: 22s total
urandom: 61s total

Volker Siegel
  • 1,504
  • 11
  • 21
  • I am using CentOS 6.5 and sharand is not available. I tried installing using: yum install sharand. It gives me "No package sharand available." Also if i run "time sharand a 1000000000", it says : command not found – egeek Aug 05 '14 at 20:36
  • Oh, I just found that the program originally was not called `sharand`, as in ubuntu, but `sharnd` whith one "a" less. So that may be just a different package name. Looking at the home page of the software, it seems like he does not provide any packages, except the source; But most tools are just an algorithm in a single .c file, and very simple to build. If you can not find a package with the original name either, we'll find another way. (Sources and math papers here: http://mattmahoney.net/dc/) – Volker Siegel Aug 06 '14 at 01:16
  • Unfortunately,i am unable to figure out how to work with sharnd also. One method bewlo with the openssl is working fine for now but only issue is how to specify the output file size. – egeek Aug 06 '14 at 19:36
4

You want a special file in Linux, /dev/random serves as a random number generator on a Linux system. /dev/random will eventually block unless your system has a lot of activity, /dev/urandom in non-blocking. We don't want blocking when we're creating our files so we use /dev/urandom.

try this command:

dd if=/dev/urandom bs=1024 count=1000000 of=file_1GB conv=notrunc

This will create a file with bs*count random bytes, in our case 1024*1000000 = 1GB. The file will not contain anything readable, but there will be some newlines in it.

xKon@xK0n-ubuntu-vm:~/tmp$ dd if=/dev/urandom of=file.txt bs=1048576 count=100 conv=notrunc
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 13.4593 s, 7.8 MB/s
xKon@xK0n-ubuntu-vm:~/tmp$ wc -l file.txt
410102 file.txt

You can use the option seek with dd to speed up the process a little more:

$ dd if=/dev/zero of=1g.img bs=1 count=0 seek=1G
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 8.12307 s, 132 MB/s
$ ls -lh t
-rw-rw-r-- 1 xK0n  xK0n  1.1G 2014-08-05 11:43 t

The disadvantages here are the fact that the file does not contain anything readable and the fact that it is quite a bit slower than the /dev/zero method (around 10 seconds for 100Mb).

You may also like fallocate command that Preallocates space to a file.

fallocate -l 1G test.img

output

-rw-r--r--. 1 xK0n xK0n 1.0G Aug 05 11:43 test.img

xxbinxx
  • 156
  • 3
2

Because there was no such tool, I have created a multi-threaded fast random data generator.

pip install fastrandom
fastrandom > /dev/yourdisk

OpenSSL encryption as mentioned in other answers is among the fastest ways to create random data but it will only use one CPU core to do so. I am able to get about 3 GB/s with either aes-256-ctr or chacha20, while my own tool goes to about 4.5 GB/s on real devices (11 GB/s to /dev/null).

openssl enc -chacha20 -nosalt -kfile /dev/urandom -in /dev/zero \
| dd of=/dev/yourdisk bs=1M status=progress

Interestingly enough, encryption appears to be the fastest way to produce random numbers. Even the best modern random number generators such as PCG64 cannot run that fast (I get about 1 GB/s per core). Both methods produce perfectly good pseudo random numbers, so it makes sense to simply use encryption for all PRNG needs.

Tronic
  • 141
  • 3
0

QUICK AND DIRTY AND FASTEST way (without installing extra utilities) is simply:

cat /dev/urandom > random.bin

In another terminal window, just repeatedly use

ls -lh

to monitor the size of random.bin. When random.bin reaches the size you want, reurn to the original window and hit Ctrl+c

I tried several of the above methods simultaneously in several terminal windows on the the same machine (a wanted to fill a 2TiB disk with random data) and this was the last I chose, and it caught up to and passed all the other methods very quickly - approx factor of 2 or 3 times as fast!

EVEN FASTER - if you are happy with zeros rather than random data, use

cat /dev/zero > fileofzeros.bin

rednectar
  • 31
  • 4