CLOUDSTACK-9746 system-vm: logrotate config causes critical failures* rotate both daily and by size by using maxsize in stead of size
* decrease the max size to 10M for rsyslog files
* remove delaycompress for rsyslog files
* increase rotate to 10 for cloud.log
* pr/1915:
CLOUDSTACK-9746 system-vm: logrotate config causes critical failures
Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
* rotate both daily and by size by using maxsize in stead of size
* decrease the max size to 10M for rsyslog files
* remove delaycompress for rsyslog files
* increase rotate to 10 for cloud.log
CLOUDSTACK-8908 After copying the template charging for that template is getting stoppedThis is happening as the zone id is not part of the query. Zone id is added to the query and unit tests are also added
* pr/896:
CLOUDSTACK-8908 After copying the template charging for that template is stopped
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9594: API "list templates templatefilter=all" reveals allAPI "list templates templatefilter=all" reveals all templates.
Using a "list templates templatefilter=all" API call any domain admin can see all templates of all domains in ACS. Information returned includes the account and domain of the template's owner.
The template data shows what that VM is using and any hints from the label. This would give an advantage in what attack vectors to use. The account and domain can possibly be used in brute force attack to guess the password and login information.
Test Scenario:
created two accounts in different domain.
```
mysql> select account_id,username,api_key from user where id in (4,5);
+------------+-----------+----------------------------------------------------------------------------------------+
| account_id | username | api_key |
+------------+-----------+----------------------------------------------------------------------------------------+
| 4 | sudadmin1 | 3qeSuWadNzUFZ_i6c6zbwafjM3Eo0TWpkHw3En9jNsg5Ditk2N18DnbbL2quBYQ7FsdXQ8rwxbyFlE8vyUTwEg |
| 5 | sudadmin | N5uHVOrg1Ek1F1a_5OXTz4WpLG3ewHqcbPUSBjQ-2CTJdxmUe2go0S8fyqH4Np0scYiehYg2KqthZXCWEyKx1A |
+------------+-----------+----------------------------------------------------------------------------------------+
2 rows in set (0.00 sec)
mysql> select account_name,domain_id from account where id in (4,5);
+--------------+-----------+
| account_name | domain_id |
+--------------+-----------+
| sudadmin | 2 |
| sudadmin1 | 3 |
+--------------+-----------+
2 rows in set (0.00 sec)
```
User sudadmin registered a private template named 'Debian'.
http://10.147.59.107:8080/client/api?apikey=N5uHVOrg1Ek1F1a_5OXTz4WpLG3ewHqcbPUSBjQ-2CTJdxmUe2go0S8fyqH4Np0scYiehYg2KqthZXCWEyKx1A&command=listTemplates&templatefilter=self&signature=ODt7zEWCLL20z1FT%2FIkd1molRaM%3D
listTemplate with "templatefilter=self", lists the newly registered template.
```
<listtemplatesresponse cloud-stack-version="4.8.0">
<count>1</count>
<template>
<id>51026d32-60ee-4e25-8ffd-3fa3c57fc14c</id>
<name>Debian</name>
<displaytext>Debian</displaytext>
<ispublic>false</ispublic>
<created>2016-11-10T17:18:00-0500</created>
<isready>true</isready>
<passwordenabled>false</passwordenabled>
<format>VHD</format>
<isfeatured>false</isfeatured>
<crossZones>false</crossZones>
<ostypeid>38c1fc84-a687-11e6-a8c8-06f654000053</ostypeid>
<ostypename>Debian GNU/Linux 7(64-bit)</ostypename>
<account>sudadmin</account>
<zoneid>25fa5b74-d4c2-4bad-8e3a-ceffcd10985e</zoneid>
<zonename>z1</zonename>
<status>Download Complete</status>
<size>2621440000</size>
<templatetype>USER</templatetype>
<hypervisor>XenServer</hypervisor>
<domain>SUDDOMAIN</domain>
<domainid>a350c00d-4048-4876-ae09-74ad4b7bb28c</domainid>
<isextractable>false</isextractable>
<checksum>e87a6d7291b999c92baa9623c9c3c207</checksum>
<details>{hypervisortoolsversion=xenserver61}</details>
<sshkeyenabled>false</sshkeyenabled>
<isdynamicallyscalable>false</isdynamicallyscalable>
</template>
</listtemplatesresponse>
```
User: sudadmin1
listTemplate with "templatefilter=self" does not list any template.
http://10.147.59.107:8080/client/api?apikey=3qeSuWadNzUFZ_i6c6zbwafjM3Eo0TWpkHw3En9jNsg5Ditk2N18DnbbL2quBYQ7FsdXQ8rwxbyFlE8vyUTwEg&command=listTemplates&templatefilter=self&signature=RfKsdg3RxDkqJotbTlHU2RdbdPA%3D
`<listtemplatesresponse cloud-stack-version="4.8.0"/>
`
NO TEMPLATES
**listTemplate with "templatefilter=all" lists all templates**
http://10.147.59.107:8080/client/api?apikey=3qeSuWadNzUFZ_i6c6zbwafjM3Eo0TWpkHw3En9jNsg5Ditk2N18DnbbL2quBYQ7FsdXQ8rwxbyFlE8vyUTwEg&command=listTemplates&templatefilter=all&signature=l5tubfyABT67d1jY702dvtZODbc%3D
Result:
```
<listtemplatesresponse cloud-stack-version="4.8.0">
<count>3</count>
<template>
<id>38451a02-a687-11e6-a8c8-06f654000053</id>
<name>CentOS 5.6(64-bit) no GUI (XenServer)</name>
<displaytext>CentOS 5.6(64-bit) no GUI (XenServer)</displaytext>
<ispublic>true</ispublic>
....
</template>
<template>
<id>51026d32-60ee-4e25-8ffd-3fa3c57fc14c</id>
<name>Debian</name>
<displaytext>Debian</displaytext>
<ispublic>false</ispublic>
<created>2016-11-10T17:18:00-0500</created>
<isready>true</isready>
<passwordenabled>false</passwordenabled>
<format>VHD</format>
<isfeatured>false</isfeatured>
<crossZones>false</crossZones>
<ostypeid>38c1fc84-a687-11e6-a8c8-06f654000053</ostypeid>
<ostypename>Debian GNU/Linux 7(64-bit)</ostypename>
**<account>sudadmin</account>**
<zoneid>25fa5b74-d4c2-4bad-8e3a-ceffcd10985e</zoneid>
<zonename>z1</zonename>
<size>2621440000</size>
<templatetype>USER</templatetype>
<hypervisor>XenServer</hypervisor>
<domain>SUDDOMAIN</domain>
<domainid>a350c00d-4048-4876-ae09-74ad4b7bb28c</domainid>
<isextractable>false</isextractable>
<checksum>e87a6d7291b999c92baa9623c9c3c207</checksum>
<details>{hypervisortoolsversion=xenserver61}</details>
<sshkeyenabled>false</sshkeyenabled>
<isdynamicallyscalable>false</isdynamicallyscalable>
</template>
<template>
<id>5f6af7bb-d965-4b9b-ab45-6d455b0d6bbe</id>
<name>SystemVM Template (XenServer)</name>
<displaytext>SystemVM Template (XenServer)</displaytext>
<ispublic>false</ispublic>
.....
</template>
</listtemplatesresponse>
```
**After Fix:**
http://10.147.59.107:8080/client/api?apikey=3qeSuWadNzUFZ_i6c6zbwafjM3Eo0TWpkHw3En9jNsg5Ditk2N18DnbbL2quBYQ7FsdXQ8rwxbyFlE8vyUTwEg&command=listTemplates&templatefilter=all&signature=l5tubfyABT67d1jY702dvtZODbc%3D
```
<listtemplatesresponse cloud-stack-version="4.8.0">
<count>1</count>
<template>
<id>38451a02-a687-11e6-a8c8-06f654000053</id>
<name>CentOS 5.6(64-bit) no GUI (XenServer)</name>
<displaytext>CentOS 5.6(64-bit) no GUI (XenServer)</displaytext>
<ispublic>true</ispublic>
<created>2016-11-10T09:32:44-0500</created>
<isready>true</isready>
<passwordenabled>false</passwordenabled>
<format>VHD</format>
<isfeatured>true</isfeatured>
<crossZones>true</crossZones>
<ostypeid>38a2bfd6-a687-11e6-a8c8-06f654000053</ostypeid>
<ostypename>CentOS 5.6 (64-bit)</ostypename>
<account>system</account>
<zoneid>25fa5b74-d4c2-4bad-8e3a-ceffcd10985e</zoneid>
<zonename>z1</zonename>
<size>21474836480</size>
<templatetype>BUILTIN</templatetype>
<hypervisor>XenServer</hypervisor>
<domain>ROOT</domain>
<domainid>383e0ea6-a687-11e6-a8c8-06f654000053</domainid>
<isextractable>true</isextractable>
<checksum>905cec879afd9c9d22ecc8036131a180</checksum>
<sshkeyenabled>false</sshkeyenabled>
<isdynamicallyscalable>true</isdynamicallyscalable>
</template>
</listtemplatesresponse>
```
Bug has been fixed considering below points
1. templatefilter=all or isofilter=all is applicable only to admin and domain admin.
2. With templatefilter=all or isofilter=all below are the visiblity of templates in system.
- admin should be able to see all templates/iso in system.
- domain admin should be able to see all public template and templates under its domain tree (including sub domain).
- domain admin in a project context should be able to see all public templates and templates registered
as project account and templates which are shared(using updateTemplatePermission api) with project account.
Also Modified "test/integration/component/test_escalation_listTemplateDomainAdmin.py"
This marvin test was written for this scenario but for the second account "templatefilter=all" is not used.
* pr/1763:
CLOUDSTACK-9594: reverted changes introduced in CLOUDSTACK-9376
CLOUDSTACK-9594: API "list templates templatefilter=all" reveals all templates of all domains
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Cloudstack 9586: When using local storage with Xenserver prepareTemplate does not work with multiple primary storeThe race condition will happen whenever there are multiple primary storages and the CS tries to mount the secondary store to xenserver host simultaneously.
Due to synchronised block one mount will be successful and other thread will get the already mounted SR. Without the fix the two thread will try to mount it parallely and one will fail on Xenserver.
* pr/1765:
Cloudstack 9586: When using local storage with Xenserver prepareTemplate does not work with multiple primary store The race condition will happen whenever there are multiple primary storages and the CS tries to mount the secondary store to xenserver host simultaneously. Due to synchronised block one mount will be successful and other thread will get the already mounted SR. Without the fix the two thread will try to mount it parallely and one will fail on Xenserver.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9635: fix test_privategw_acl.pyensure VLAN used for createPrivateGateway is determined after the guest
networks in the VPC is created, so that we skip VLAN allocated for guest
network for the private network of vpc gateway
* pr/1802:
CLOUDSTACK-9635: fix test_privategw_acl.py
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
ensure VLAN used for createPrivateGateway is determined after the guest
networks in the VPC is created, so that we skip VLAN allocated for guest
network for the private network of vpc gateway
The race condition will happen whenever there are multiple primary storages and the CS tries to mount the secondary store to xenserver host simultaneously.
Due to synchronised block one mount will be successful and other thread will get the already mounted SR. Without the fix the two thread will try to mount it parallely and one will fail on Xenserver.
This fixes build_asf.sh release script to update checkstyle pom.xml with the
provided new version.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
fix marvin test failure test_router_dhcp_optsmarvin, VirtualMachine object's, nic attribute does not have nic's in any
particualr order in the array. so check isdefault attribute to the get non-default nic
earlier test made assumption that nic[0] is default nic, which is not true always
* pr/1801:
CLOUDSTACK-9634: fix marvin test failure test_router_dhcp_opts
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
marvin, VirtualMachine object, nic attribute does not have nic's in any
particualr order in the array. soi check isdefault attribute to the get non-default nic
earlier test made assumption that nic[0] is default nic, which is not true always
CLOUDSTACK-9566 instance-id metadata for baremetal VM returns IDThere is difference in instance-id metadata across baremetal and other hypervisors.
On Baremetal
[root@ip-172-17-0-144 ~]# curl http://8.37.203.221/latest/meta-data/instance-id
6021
on Xen
[root@ip-172-17-2-103 ~]# curl http://172.17.0.252/latest/meta-data/instance-id
cbeb517a-e833-4a0c-b1e8-9ed70200fbbf
In both cases it should be vm's uuid.
* pr/1738:
CLOUDSTACK-9566 instance-id metadata for baremetal VM returns ID
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9561 After domain/account deletion, snapshot taken by the
domain/account remains undeleted
While deleting the UserAccount Cleanup for the removed VMs/volumes are not
happening. For the removed VMs, snapshots doesn't get cleaned. Only for running
VMs(volumes in ready state) the cleanup happens.
When the VM is desroyed, the volume is marked as destroyed and later storage
garbage collector perform the cleanup. But if we try delete domain/account
before storage garbage collector runs, then it fails.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9598: wrong defaut gateway for the nic in non-default network when guest VM has nic's in more than one guest network set the tag for each host in /etc/dhcphosts.txt, and use the tag to add exception in /etc/dhcpopts.txt to prevent sending default route, dns server in case if the nic is in non-default network
this was the behaviour with edithosts.sh prior to 4.6
* pr/1766:
CLOUDSTACK-9598: wrong defaut gateway for the nic in non-default network
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9503: Increased the VR script timeout. Most of the changes are about converting int/long time values to joda Duration.
* pr/1745:
CLOUDSTACK-9503: Increased the VR script timeout. Most of the changes are about converting int/long time values to joda Duration.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9583: VR: In CsDhcp.py preseed both hostaname and localhost to resolve to 127.0.0.1
The VR executes a ip route flush command as part of configurations. This command performs a
DNS lookup on the VR hostname. Since the VR does not have a DNS entry, the ip command would
wait 5 seconds before timing out and executing the flush operation. This fix adds the VR
hostname to /etc/hosts mapped to 127.0.0.1 to answer the DNS lookup reducing the
execution time.
* pr/1757:
CLOUDSTACK-9583: VR: In CsDhcp.py preseed both hostaname and localhost to resolve to 127.0.0.1
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
when guest VM has nic's in more than one guest network set the tag for
each host in /etc/dhcphosts.txt, and use the tag to add exception in
/etc/dhcpopts.txt to prevent sending default route, dns server in case if the nic is in non-default network
this was the behaviour with edithosts.sh prior to 4.6
added new test case test_router_dhcp_opts to test DHCP option file use of cloudstack
CLOUDSTACK-9460: For long running transactions, if the connection istimed out by the mysql server then refresh it
* pr/1674:
CLOUDSTACK-9460: For long running transactions, if the connection is timed out by the mysql server then refresh it
Signed-off-by: John Burwell <meaux@cockamamy.net>
CLOUDSTACK-9071: Properly parse stats.output.uri in StatsCollectorBoth host and path could have been NULL which causes the StatsCollector
no to start properly.
By checking if the Strings are not Empty or Null we make sure the StatsCollector
always runs and does not prevent the Management Server from starting.
* pr/1673:
CLOUDSTACK-9071: Properly parse stats.output.uri in StatsCollector
Signed-off-by: John Burwell <meaux@cockamamy.net>
templates of all domains
Bug has been fixed considering below points
1. templatefilter=all or isofilter=all is applicable only to admin
and domain admin.
2. With templatefilter=all or isofilter=all below are the visiblity
of templates in system.
a. admin should be able to see all templates/iso in system.
b. domain admin should be able to see all public template and
templates under its domain tree (including sub domain).
c. domain admin in a project context should be able to see all public
templates and templates registered as project account and templates
which are shared(using updateTemplatePermission api) with project account.
Modified
"test/integration/component/test_escalation_listTemplateDomainAdmin.py"
This marvin test is written for this scenario but for the second account
"templatefilter=all" is not used.
The VR executes a ip route flush command as part of configurations. This command performs a
DNS lookup on the VR hostname. Since the VR does not have a DNS entry, the ip command would
wait 5 seconds before timing out and executing the flush operation. This fix adds the VR
hostname to /etc/hosts mapped to 127.0.0.1 to answer the DNS lookup – reducing the
execution time.
CLOUDSTACK-9183: bash: /opt/cloud/bin/getRouterAlerts.sh: No such file or directory
* pr/1744:
CLOUDSTACK-9183: bash: /opt/cloud/bin/getRouterAlerts.sh: No such file or directory
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9553 Usage event is not getting recorded for snapshots in a specific scenario
* pr/1714:
CLOUDSTACK-9553 Usage event is not getting recorded for snapshots in a specific scenario
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9554 Juniper Contrail plug-in is publishing events to wronJuniper Contrail plug-in is publishing events to message bus instead of event bus
* pr/1715:
CLOUDSTACK-9554 Juniper Contrail plug-in is publishing events to wrong message bus
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9509: Host Connects Without StorageKVM hosts on shared storage failure was accepted by mgmt server with the
host state as Up, even though there was no primary/shared storage available on
it. This patch offers a quick fix by throwing an exception in the storage monitor
which connects storage pool on host. The failure is trapped by agent manager
that disconnects the agent without any investigation.
Based on Lab tests, KVM agent may take upto 2 minutes to attempt NFS mount when
the storage is inaccessible (firewalled, or shutdown) before returning back with
an error. It is safe to assume that this won't add pressure on mgmt server due to
several reconnection attempts, and KVM agent would retry reconnection every 2
minutes.
For such KVM hosts, where failure happens due to storage issues; they will be
briefly put in Alert state but will be mostly be in Connecting state during which
the KVM host attempts to mount/reconfigure NFS storage pool.
/cc @jburwell @karuturi
@blueorangutan package
* pr/1694:
CLOUDSTACK-9509: Host Connects Without Storage
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9534: Allow VR to be destroy in UI when in running stateThis allows us to destroy a VR that is in running state from the UI. This action is possible via cloudmonkey/APIs, and would be useful for anyone who wants to simply destroy a VR from the UI without having to stop it first.
As this is a pure UI fix, manual testing would be needed but no regression integration testing is necessary.
/cc @jburwell @karuturi
@blueorangutan package
Screenshot:
* pr/1701:
CLOUDSTACK-9534: Allow VR to be destroy in UI when in running state
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9550: Use context to filter items in a metrics viewUse available context to filter a metrics view based on zone, cluster, host
in the context object. This fixes metrics view filtering when metrics view is
viewed via Zone->Compute and Storage-> for a resource.
/cc @jburwell @karuturi -- this is a pure UI fix, one manual test LGTM would be required.
* pr/1712:
CLOUDSTACK-9550: Use context to filter items in a metrics view
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9511: fix test_privategw_acl.py to handle multiple physical network
fix to ensure only physical network with guest traffic is picked up for
creating a private network for vpc private gateway
* pr/1724:
CLOUDSTACK-9511: fix test_privategw_acl.py to handle multiple physical network
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
fix to ensure only physical network with guest traffic is picked up for
creating a private network for vpc private gateway
Signed-off-by: Murali Reddy <muralimmreddy@gmail.com>
This closes#1724
Fixes regarding VOLUME_DELETE events resulting from account deletionNew version of #1491.
**Original Description**
New version of #1373, but updated for the 4.7 branch with another fix that allows it to properly find expunged root volumes. This is a bug fix, which is why we target the 4.7 branch.
Original pull request:
Fixes regarding usage event emission.
UsageEventUtils was previously not checking deleted accounts, which meant that if an account was deleted that had some resources running on it, those resources would get destroyed without emitting any events.
Furthermore, the VOLUME_DELETE event of ROOT volumes is the responsibility of the UserVmManager, which gets circumvented when expunging resources following the account deletion. Added a check to the AccountManager which catches the ROOT volumes that need to be deleted and emits events for them.
To test this: Create a new user. As that user, create and destroy an instance. This should cause the VM_CREATE, VM_START, VM_STOP, VM_DESTROY, VOLUME_CREATE, and VOLUME_DELETE events to be emitted.
Create a new instance as the same user. Log in as admin, and delete the user. The same set of events should be emitted, and there should be no duplicate DELETE events for the ROOT volume of the previous instance.
* pr/1624:
Emit a VOLUME_DELETE usage event when account deletion destroys an instance.
Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
Rajani Karuturi
Joakim Sernbrant
Rohit Yadav
Sudhansu
Murali Reddy
Abhinandan Prateek
Koushik Das
John Burwell
subhash yedugundla
Ólafur Stefán Arnarsson
Marc-Aurèle Brothier
Sudharma Jain
subhash_y
