Why are they not enough for your needs? Also, why do you need to optimize that much (remember that your developer's time costs more than the computer your program is running on).Īre you interested in splitting a thousand of files per day of a few gigabytes each, or splitting a dozen of files per day each of at least hundreds of gigabytes? These are two different problems! (I assume you have some ordinary desktop). And it seems that you are reinventing csplit(1) or split(1). When used properly, they could improve overall performance. if most files have a few gigabytes) things could be different.īTW, on Linux, you might be interested by system calls like posix_fadvise(2) and/or readahead(2). If the files are not huge and could entirely fit in the page cache (e.g. So your benchmarks won't be exactly reproducible.Īt last, your problem (splitting huge files of hundred of gigabytes each) is probably disk-IO-bound, not CPU bound, so the actual way of coding should not matter that much, at least if your buffers have suitable sizes (at least 128 kilobytes, and more likely a few megabytes see setvbuf(3).). Don't expect your system to be deterministic and to give the same timings for several runs. When you split a huge file, it is likely to have been generated (or downloaded, or obtained) a few seconds or minutes ago (why would you wait several hours before splitting it), so you really care more about a "warm" state, and in practice is it likely to be (partly) in your page cache.ĭetails are obviously computer, operating system, and file system specific. I dont think that measuring a "cold" state is realistic in your case. in the page cache) when you would really use your program. In practice, it is very likely that some of the data is already "here" (e.g. In your case, I believe you want to consider the average time. Read also Operating Systems: three easy pieces for more about OSes. Hence the kernel scheduler is behaving differently from one run to the next (because of preemptive scheduling.
#RAM BENCHMARK SOFTWARE SOFTWARE#
and perhaps CPU frequency -limited when the chip is too hot- is changing without software control. Remember that your hardware is non-deterministic: CPU cache behavior, CPU pipelining, superscalar processors with out-of-order execution, external interrupts -timers, networks, USB, disk. Also "starting" or "cold-start" operation is not a typical running condition (but a special case), so you usually want to ignore it. Hence, you need to make several benchmarks. In general, you cannot benchmark any software in exactly the same conditions because computer (and their operating systems) are not entirely deterministic, so you won't be able to reproduce exactly some running conditions. You could choose the worst one (probably the first time), or you could consider an average of them, or ignore the worst and best runs and only care about the rest of them, etc. The next question is what timing is the most relevant. run five times exactly the same thing) the same "solution" and benchmark them all. Run the software inside a docker container, each time.
#RAM BENCHMARK SOFTWARE PC#
Please provide me some info as I am researching a lot and could not find any helpful resource. Then I can calculate the average speed of a, the average speed of b and finally compare the data. Run a.exe and measure time, clean all ram, cpu register anything cached about this data What is the best way to make this kind of tests ? I wonder if there is any way to do it without having to restore the PC? If I restart the PC and measure performance again, then the ram is empty from the previous data and results are quite ok. I want to run each test with a clean slate. I want to make the tests as more objective and reproducible as possible by removing these influences. If I run more then once the same solution, the execution time will be less and I understand that this happens because some of the data gets cached in the memory hierarchy (ram, or cpu registers etc). (with threads, go routines, MPI etc) and want to objectively compare them. I am measuring the execution time of each of my solutions. I am writing a software that splits big files into smaller files and I have coded several solutions.
0 kommentar(er)
