haproxy tunning:
0. Test case:
httpd running in 5 user VMs, all of them created on a xenserver host(16 core, 42G memroy, 10G network)
domR running on an anther host with same hardware configuration.
test application, ab, running on anther host behind an anther seperate switch
1.haproxy is not a memory intensive app. I can get 4625.96 connection/s with 1G memory. While it's really a CPU intensive app, domR always uses around 100% CPU on the host.
2.By default, you can't get better connection/s rate, because ip_conntrack_max and tw_bucket are too small, you will see the error in domR like:
"TCP: time wait bucket table overflow" or "nf_conntrack: table full, dropping packet".
So I increase these numbers to 1000000 from 65536, then I can steadly get around 4600 connection/s when memory is >= 1G.
Here is the connection per second, tested by "ab -n 1000000 -c 100 http://192.168.170.152:880/test.html"
domR memory conn/s
128M: 3545.55
256M: 4081.38
512M: 4318.18
1G: 4625.96
7G: 4745.53
3. If I enable notrack for both connections between domr/user vm, and public network, that tell iptable in domR don't track the connection during my test, then I can get better number, around
5800 connections/s. But we can't enable notrack, as iptables is used to track throughput in domR.
4. In a word, with this commit, the connection rate of haproxy can be increased from 1000-2000/s to 4700/s when domR's memory is larger than 1G.
5. How many CPU need to assign to domR to get this number? Haven't finished yet, as CPU is shared by all the VMs on the host, if other VMs are busy, it will impact the performance of haproxy.
Also fixed a couple of other problems:
* verify security group ids before vm creation
* don't create "default" security group (if missing) as a part of deployVm process when vm is deployed from vmWare template
fixed a DB Exception
5-ThreadPoolExecutor.runWorker:1110-ThreadPoolExecutor$Worker.run:603; called by -Transaction.rollback:783-Transaction.removeUpTo:726-Transaction.close:549-DatabaseCallback.interceptComplete:72
2011-05-19 19:02:41,242 WARN [utils.nio.Task] (AgentManager-Handler-1:null) Caught the following exception but pushing on
com.cloud.utils.exception.CloudRuntimeException: DB Exception on: com.mysql.jdbc.JDBC4PreparedStatement@4faae357: INSERT INTO usage_event (usage_event.type, usage_event.created, usage_event.account_id, usage_event.zone_id, usage_event.resource_id, usage_event.resource_name, usage_event.offering_id, usage_event.template_id, usage_event.size, usage_event.resource_type, usage_event.processed) VALUES (_binary'TEMPLATE.CREATE', '2011-05-20 02:02:41', 2, 2, 202, _binary'anth-te', null, 2, -1, null, 0)
at com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1218)
at com.cloud.event.dao.UsageEventDaoImpl$$EnhancerByCGLIB$$377dce70.CGLIB$persist$22(<generated>)
at com.cloud.event.dao.UsageEventDaoImpl$$EnhancerByCGLIB$$377dce70$$FastClassByCGLIB$$30e2aaaf.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:215)
at com.cloud.utils.db.DatabaseCallback.intercept(DatabaseCallback.java:35)
at com.cloud.event.dao.UsageEventDaoImpl$$EnhancerByCGLIB$$377dce70.persist(<generated>)
at com.cloud.storage.download.DownloadMonitorImpl.handleDownloadEvent(DownloadMonitorImpl.java:410)
at com.cloud.storage.download.DownloadListener.setDownloadInactive(DownloadListener.java:358)
at com.cloud.storage.download.DownloadCompleteState.onEntry(DownloadCompleteState.java:44)
at com.cloud.storage.download.DownloadListener.transition(DownloadListener.java:240)
at com.cloud.storage.download.DownloadListener.processAnswers(DownloadListener.java:224)
at com.cloud.agent.manager.AgentAttache.processAnswers(AgentAttache.java:258)
at com.cloud.agent.manager.AgentManagerImpl$AgentHandler.processResponse(AgentManagerImpl.java:2319)
at com.cloud.agent.manager.AgentManagerImpl$AgentHandler.doTask(AgentManagerImpl.java:2334)
at com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:537)
at com.cloud.utils.nio.Task.run(Task.java:85)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: com.mysql.jdbc.MysqlDataTruncation: Data truncation: Out of range value for column 'size' at row 1
status 9873: resolved fixed
Following fixes were made as a part of the checkin:
* When deploy user vm and SG doesn't exist in the DB, create it automatically.
* SecurityGroup enabled use vm start: if map to default group is not present in security_group_vm_map table, create one.
* Added "name" (securityGroupName) parameter back to deleteSecurityGroup/authorizeSecurityGroupIngress/deployVm. Mutually exclusive with security group id parameter.
Conflicts:
api/src/com/cloud/api/commands/AuthorizeSecurityGroupIngressCmd.java
api/src/com/cloud/api/commands/DeleteSecurityGroupCmd.java
api/src/com/cloud/api/commands/DeployVMCmd.java
server/src/com/cloud/api/ApiDBUtils.java
server/src/com/cloud/vm/UserVmManagerImpl.java
Edison Su
Alex Huang
alena
Abhinandan Prateek
anthony
Chiradeep Vittal
Jessica Tomechak
Murali Reddy
kishan
Kelven Yang
Jessica Wang