LoadSim

LoadSim is now finished.

LoadSim creates different types of synthetic workloads on a linux system. It consists of a script that calls upon different programs to perform the workloads.

Its dependencies are curl, and lookbusy (which can be downloaded from
http://devin.com/lookbusy/download/lookbusy-1.4.tar.gz ).

CPU and memory workload is provided by lookbusy, network workload is provided by curl and downloading and uploading a file to a ftp server, and the disk read and write is done by a C program that I have written.

CPU workload
Lookbusy looks in /proc/cpuinfo to attempt to find how many CPU cores the current system has. If more than 1 CPU is found, the process forks itself and each process performs its load on a separate CPU each. (The number of CPUs can be specified directly if needed).

The workload is given as a percentage of CPU to be used by all cores. The workload is then generated by performing many addition equations in succession. So this creates a workload, but then Lookbusy continuously monitors /proc/stat for changes in CPU usage, attempting to keep total system CPU load at the desired level, sleeping the process of calculations when it goes over the specified level, and resuming when it goes under.










Memory workload
Lookbusy is also used to perform the memory workload. The memory that is allocated is continually stirred to ensure it is actually allocated in the virtual memory system and imposes pressure on it to keep it resident. In other words, it keeps the memory in use, not just allocate it and let it sit there doing nothing. It has to do this because of the way Linux uses virtual memory. If the allocated memory is not continuously stirred, Linux marks it as low priority and may be discarded later if other needs arise.

The memory workload is generated by using the memcopy() function in C to copy bytes of data between memory blocks. It takes 3 arguments, destination, source and count. The arguments destination and source point to the destination and source memory blocks respectively. count specifies the number of bytes to be copied.

If, however, the memmove() function is available, it will use that. Memmove() does everything memcopy() does, but also handles overlapping memory blocks. With memcopym if the two blocks of memory overlap, the function might not operate properly --some of the data in source might be overwritten before being copied. Memmove() ensures that the source data in the overlapped region is copied before being overwritten.

However, in our case, we do not care if data is overwritten or not in the memory only that a workload is performed, so it matters little to me which one of these functions is used.

Similar to the CPU workload, lookbusy will also alternate between copying and sleeping to keep the workload at the desired level.

Network workload
Network workload was produced by a command line application called curl, which is written in C. Curl is used for transferring data to and from a server using various protocols. Both downloading and uploading is supported by it.

Curl uses a library called libcurl for all its functions. It is portable and works on alarge range of operating systems, and has its own API for other programs to call upon. I chose curl, as it was written in C and many Linux distributions come with it by default, That way, the dependencies are minimised.

Curl was chosen as it allowed the bandwidth to be limited during a transfer, with the limit rate option. This was used alongside a FTP server which I had setup to download and upload to. The virtual machines are loaded with some large garbage files to do this. Curl does not have resume download support, which means that each time we execute these commands, they start redownloading or reuploading the file, which is a good thing in my case.

The FTP server does not use encryption, as it would introduce extra unwanted CPU overhead into the virtual machines.

DISK READ WRITE DESCRIPTION
I also managed to get disk read and write working, however it is not very stable (more on this later).

For disk read, I had attempted to create a file with lines filled with randomly generated 1MB long strings on each line, approximately 2000 lines for a 2GB file. I then used fopen(), and fread() to read from the file (into the memory), telling it to read the specified amount of megabytes in the file, then waiting for 1 second (to create a disk read workload of megabytes per second).

Same principle was applied for disk write. A file was opened and read into memory, then the file was written from memory into a new file.

Disk read and write is not very stable when used together at speeds greater then ~13MB/s. Disk read slows down to about 2MB/s and disk write continues at its own workload.
-------------------------

The LoadSim Script
The LoadSim script calls upon these programs to generate a workload. It accepts and parses command line arguments given to it in order to tell these applications how much load it should perform.

I used getopts() to parse through the command line arguments. It calls upon lookbusy twice to separately perform CPU and memory workloads.

Curl is also called upon twice, first time which is to download a remote file, and 2nd time to upload a local file to the FTP server.

There was also a slight hitch with lookbusy.If you specified an amount of CPU or memory to be used, lookbusy will use that amount of CPU or memory, and not a create a workload UP TO that level. For example, if my system was already running with 300MB of memory in use, and I specified to lookbusy that I wanted a workload of 500MB, then lookbusy itself will end up using 500MB, and my total memory usage will be 800MB. I solved this in my script by calculating the current resource usage, and then subtracting it from the amount specified. that way, if my system was running with 300MB, and I specify a workload of 500MB, then lookbusy will perform a workload of 200MB to make a total of 500MB of memory used.
-----------------------

The scripts can be found on this link:
https://www.sendspace.com/file/nmd5rc

Progress Report

For this semester, I have written a script that calls upon other programs to create a synthetic workload on a Linux-based machine. This script can create CPU, memory and network workload, and is controllable by a command line interface. The script calls upon the following programs to create the workload:

CPU: lookbusy

MEM: lookbusy

Network upload/download: cURL

I could not get disk read/write working at a consistent rate, and so it was left out of the script.
I have also created some virtual machines (VM) for Javid to put up on the University of Sydney's servers. These VMs are for me to execute my experiments for my thesis project.
Three of these VMs are simple Ubuntu VMs which have my script and the dependencies installed.
UbuntuVM1 is on one host, UbuntuVM2 is on another. UbuntuVM3 is a iSCSI drive, which allows it to be moved from host to host.
The fourth Ubuntu server (FTPserver) acts as a FTP server for the other three VMs so they can perform the upload/download workload. FTPserver is on yet another host, to prevent its resource usage from affecting the results that will be gathered.
I have also started writing down the experiments I will perform with my VMs. There are many experiments to perform, and this is expected to take a large majority of the time.
I have not yet started collecting data yet. This will begin in the next few days.
At Javid's request, I will also make a 10-minute presentation on what I have done so far to him and a fellow student that is doing a similar project for Tuesday 10^th June. I have started making preparations for this.

Revised Proposals

Whereas early in the semester I was more focused on observing the resource utilisation of VMs as we add more VMs to a physical server, I will now be more focused on the live migration aspect of the resource utilisation issue for this thesis project.
I will be studying how workloads of the source physical machine, target physical machine, and on the live migrating VM itself will affect the time it takes to live migrate.

Revised Plan

Plan does not have to be revised.

Progress Report

Today I met with Javid to talk about what I will have to do in the following weeks.

The first thing I will have to do is the Progress Report for ELEC4718. It must contain:

1. An introduction
2. A literature survey, of 5-10 pages, in a form similar to that which will appear in the final treatise
3. A report of not more than 1000 words on the work carried out thus far
4. A revised proposal/plan for the following semester's work.

Javid has mentioned that for the literature survey, I should have to read 20-30 relevant research papers on the subject. 3-4 pages should be related to modelling and benchmarking, whilst 6-7 pages should be related to virtual machines + migration.

Each article should be reviewed in about 2-3 lines.
--------------------------
The 2nd thing I will have to which is related to this thesis work is to do more research on modelling. So far I've found 8 papers, but only 2-3 of them seem related. The papers are on linear and non-linear models.
--------------------------
The 3rd thing I have to do is fix up any more problems in my script, and to rethink some of the experiments I will have to perform. I forgot to account for the migrating virtual machine performing some load whilst migration occurs. (I had only considered the source and target host performing some load).
--------------------------
The last thing I will have to do is to prepare a 10-minute presentation for Javid and another student on how I implemented my script. I will have to talk about how I implemented, what I used, things I achieved and any problems I had.

This presentation will be in 2 weeks time, and should be approximately 10 slides long. Javid will email me more details later.
----------------------
One these things are done, I will have perform the actual experiments and collect results. Javid has suggested that I use MatLab with the results to assist in finding a model that fits the data, and to perform any graphing. Collecting data is expected to take a very long time, as there will be MANY experiments to perform.

Virtual Machines on Server

2 days ago I gave Javid my virtual machines which I had built to do my experiments on.

He uploaded them onto the server, so now I can access them on VSphere. The virtual machines are setup in the following way:

• There are 2 host machines, with at least 1 static VM on them.
• There is another host machine that is stored on another host, to ensure that it is isolated from these other 2 host machines.
• There is a 3rd VM that is an iSCSI drive, which allows it to be migrated between the 2 host machines.
• Using my program, I should be creating load on the static VMs, and migrating the 3rd VM between the hosts.

For the next 2 weeks (until my next meeting), I should be doing the following:

• Research on modelling.
• Play around with VSphere, especially with the migration option, to get the hang of using it
• Think about the experiments I will be performing.

Program nearing its end

Today I talked with Javid, showing him the script wrapper I have done so far.

CPU: lookbusy
MEM: lookbusy
Network upload/download: cURL
FileIO: none.

Currently, memory workload works fine with lookbusy (but still needs validation). Network workload works fine, but there is a problem that there is CPU overhead at speeds greater than 1 MB/s with Curl.


CPU workload works, but more calculations must be made to ensure that the CPU overhead from other processes do not interfere too much with the workload generated. The main concern is the CPU overhead from cURL. Javid suggested that I do some tests with cURL, find the amount of CPU overhead created by cURL at different rates, and subtract it from the CPU workload created from the program.

FileIO workload was done incorrectly. It has come to the point where I will have to create my own program that so be able to control disk read and write.

This program, should potentially use a timer (say, every second) and perform the read and write operations every second. For Disk read, Javid gave the example of creating a 1GB garbage file, with each line in the file being a 1MB string. The program can then randomly read multiple lines in parallel to achieve a controlled disk read speed. For example, if we wanted 5MB/s, the program should pick 5 random lines in the 1GB every second, read them, then pick another 5 random lines, and repeat. By picking lines randomly, we attempt to avoid problems created by the OS caching.

For disk write, the program should write a 1MB string in a line to a file, then flush it to make sure Linux actually writes it down, every second. This should be done in parallel for higher write speeds.

Hopefully all this should be done by the weekend. Javid wants me to finish this off by the weekend, even if not fully complete.

Afterwards, I will have to build my virtual machines, and think about the future experiments i will be conducting.

The virtual machines have to be done by next Thursday, as I will be handing them in so they can be put into the system.

Progress so far

I've managed to find a few programs to help me write up my benchmark/stress testing.

They are Stress (for stress testing), CpuLimit (to limit the amount of CPU used by a process), and netperf (to test bandwidth usage).

Stress appears to be very appropriate as it is cable to stress test CPU (and multiple cores if needed), memory, and HDD read/write speeds. Combine it with CPULimit, and I may be able to control the majority of the resources needed.

The Topic Proposal is due this Friday. I have already discussed with Javid about the schedule/plan for the year. I will have to use something like Microsoft Project to make a good looking Gantt Chart.

This weekend I will work on my on project. (These past 2 weeks I have been finishing off my internship at Arrobyte, so I've been quite busy).

The purpose of this project is to attempt to quantify the performance degradation of VMs according to their own loads as well as neighbouring VMs using shared resources on the same physical machine.

I talked with Javid on Wednesday, 5th of March. 

In order to do this project, Javid has set up a VM on the SIT servers for me, using VMWare vSphere. I have full control over this VM. Currently this VM has 2 virtual CPU's of 2GHz each, 2GB of RAM, and Disk I/O speed capable of 2GB/s. The VM is running Ubuntu 12.04. I can change how much resources my VM uses, however this should be enough to start off.

What I need to do for this project overall is to write a program that utilises the VM resources in a controlled way. The program should be able to be called upon in the command line, and must be able to change the utilisation of CPU, memory, HDD read, HDD write, bandwidth in and bandwidth out.

This program will be used on multiple VM's to test the performance degradation.

My main focus for today will be researching benchmarks and finding looking through their source codes for utilisation of CPU and memory. Javid mentioned that Disk I/O and bandwidth utilisation will be a bit more complicated to setup, so those will worked on later.

The program itself should be written in C or C++.

Some tips Javid gave me on the benchmarks:
1) CPU benchmarks may involve calculations, like multiplying numbers.
2) Memory benchmarks should use malloc.
3) Network benchmarks should utilise a FTP server to download and upload files.
4) Disk I/O benchmarks may involve moving a large file from one drive to another.