CloudBolt Software Logo

CloudBolt Blog

Create a CloudBolt Plug-in: Check EC2 Instance Reachability

Posted by Rick Kilcoyne

11/2/15 11:30 AM

UPDATE: As of Verison 5.3.1, it will no  longer be necessary to check EC2 instance reachability. This functionality has been rolled into the product. This article is still a great example of what it takes to write a CloudBolt plug-in and will be useful in many other scenarios. ~Rick

 

A common use-case I see frequently is the need to make sure new EC2 instances are up and ready to accept SSH connections before CloudBolt marks the provisioning job as complete. In this article, we’re going to work together to write a CloudBolt plug-in that will add this functionality to our CloudBolt environments. In doing so, I hope you'll not only gain an appreciation for the power of CloudBolt as a cloud automation platform, but you'll also see how easy it is to extend our base feature set using upgrade-safe scripts.

Getting Started

Writing Python code is a relatively painless process that usually starts with a text editor. I use OSX, so I prefer TextMate. If you’re a Windows user, I suggest Sublime Text 2 (http://www.sublimetext.com/2) or Notepad++. Another great option is to use PyCharm for all your CloudBolt plug-in development projects. I plan to expand on this topic in a future article.

Planning Our Attack

Let’s talk briefly about what we want to accomplish with this plug-in: When we provision a VM to EC2 via CloudBolt, we want to wait until that server is finished initializing and ready for SSH access before marking the entire CloudBolt provisioning job as complete. By default CloudBolt marks the job complete once the VM state is set to “OK” by AWS. Unfortunately, this isn’t the full story on the VM's readiness. The “OK” state is set before the VM is initialized and before the user can login via SSH. Imagine your poor users – they just used the awesome CloudBolt platform to spin up a VM, and once their job is “complete”, they get a “Connection Refused” error when they try to connect via SSH – not cool.

To address this issue, we'll extend CloudBolt to wait until our new EC2 instance has passed all EC2 status checks before marking the job as successfully completed. To accomplish this, we’ll trigger an action at the post-provision stage of the “Provision Server” Orchestration Action that will poll EC2 every two seconds to see if our new instance is reachable according to the EC2 status checks. We‘ll implement this action as a CloudBolt plug-in script written in Python.

Starting our Plug-in

Let's start our plug-in with a file called “poll_for_init_complete.py” with the following contents:


def run(job, logger=None, **kwargs):
    return """"""

The CloudBolt platform knows to call this function when it‘s time to execute the plug-in, therefore it's essential that it exists in your plug-in script. Note that the first and required parameter to this function is called job. This implies that we should expect the CloudBolt platform to call this function with the originating provisioning job passed as a job.models.Job object.

Returning a tuple of ("", "", "") is the default way of communicating to the CloudBolt platform that the script was a success.

Let's Get Busy

Let's add a few more lines to our plug-in script to get the server (our new EC2 instance) from the Job object and wait until it's reachable:


import time
from jobs.models import Job
 
TIMEOUT = 600
 
def run(job, logger=None, **kwargs):
    server = job.server_set.first()
    timeout = time.time() + TIMEOUT
 
    while True:
        if is_reachable(server):
            job.set_progress("EC2 instance is reachable.")
            break
        elif time.time() > timeout:
            job.set_progress("Waited {} seconds. Continuing...".format(TIMEOUT))
            break
        else:
            time.sleep(2)
 
    return """"""

Let's walk through what what we have so far:

server = job.server_set.first() sets the variable called server to the Server object associated with this job. Since we're working with a server provisioning job, it's safe to assume we're only going to have one Server associated with this job, therefore we call first() on our job's server_set property.

We defined a constant called TIMEOUT in our plug-in module and set it to 600. We then use this TIMEOUT at timeout = time.time() + TIMEOUT to set the time at which we should no longer wait for our EC2 instance to initialize. This prevents CloudBolt from waiting indefinitely if for some reason EC2 cannot determine the reachability of our server. Since this is in seconds, we'll stop waiting after a maximum of 10 minutes has passed before marking the job as complete. This should be the exception – not the norm.

We then start an infinite loop that will only stop when either our timeout elapses or we determine that our EC2 instance is reachable with the function is_reachable(server) which we haven't yet defined.

Is it Reachable or Not?

The script above is still missing the implementation of our is_reachable function. Given the server object associated with this job, this function will use the AWS Boto API to determine the reachability status for our new EC2 instance.  Note: Boto is the name of the Python API used to access the AWS API.

Let's add our is_reachable function to our script above our run function:


import time
 
TIMEOUT = 600
 
def is_reachable(server):
    instance_id = server.ec2serverinfo.instance_id
    ec2_region = server.ec2serverinfo.ec2_region
 
    rh = server.resource_handler.cast()
    rh.connect_ec2(ec2_region)
    wc = rh.resource_technology.work_class
 
    instance = wc.get_instance(instance_id)
    conn = instance.connection
    status = conn.get_all_instance_status(instance_id)
    return True if status[0].instance_status.details[u'reachability'] == u'passed' else False
 
 
def run(job, logger=None, **kwargs):
    # SNIP... 

Let's step through this function step-by-step:

  1. instance_id = server.ec2serverinfo.instance_id
    Get the EC2 instance ID associated with our new server being provisioned through CloudBolt. This is a string that looks like i-2423c494 in the EC2 console.

  2. ec2_region = server.ec2serverinfo.ec2_region
    Get the AWS region into which our new EC2 instance is being deployed.

  3. A few CloudBolt platform API gymnastics to get the backing Boto API objects without specifying any credentials. Always keep credentials out of your scripts!
    rh = server.resource_handler.cast()
    rh.connect_ec2(ec2_region)
    wc = rh.resource_technology.work_class

  4. instance = wc.get_instance(instance_id)
    Get the Boto Instance object associated with our new server's instance ID.

  5. status = instance.connection.get_all_instance_status(instance_id)
    Using the connection associated with our Boto Instance object, return the instance status for our server.

  6. return True if status[0].instance_status.details[u'reachability'] == u'passed' else False
    If the reachability status for our server is “passed”, return True because our new server is now reachable. If not, return False. We use status[0] because our get_all_instance_status function above returns an array. In this case we're only asking for the status of one instance, so we know the array only has one Status object and thus we use status[0].

Going back to our loop you can now see how the is_reachable function is used to keep the loop going if the answer is false:


while True:
    if is_reachable(server):
        job.set_progress("EC2 instance is reachable.")
        break
    elif time.time() > timeout:
        job.set_progress("Waited {} seconds. Continuing...".format(TIMEOUT))
        break
    else:
        time.sleep(2)

If our server is NOT reachable, and our timeout hasn't expired, we wait two seconds and try again.

Putting it All Together

The complete script can be downloaded from cloudbolt-forgeCloudBolt Forge is a source of user-contributed actions and plug-ins

Now that it's ready, let's add it to the appropriate trigger point in CloudBolt.

In your CloudBolt instance, navigate to Admin > Actions > Orchestration Actions and click “Provision Server” on the left tab bar. Find the “Post-Provision” trigger point at the bottom of the page and click the “Add an Action” button.

Select “CloudBolt Plug-in” and in the next dialog, click "Add new cloudbolt plug-in".

Specify a name for our new plug-in (Poll for EC2 Init Complete), select the "Amazon Web Services" resource technology, browse to your script, and click "Create".  Selecting the "Amazon Web Services" resource technology ensures this plug-in only runs against AWS resource handlers that you've defined and not others to which this plug-in is not applicable.

Give it a try

Provision a server to one of your AWS-backed CloudBolt environments. Watching the job progress, you'll see that the job is not marked as complete until the server is fully reachable and SSH access is available.

Questions? Comments? Concerns?

Don't hesitate to reach out to me (rkilcoyne@cloudbolt.io) or any of the CloudBolt Solutions team for help!

Read More

Topics: Automation, AWS, CloudBolt

App and Cloud Management, added Nebula Private Cloud with v4.6

Posted by Justin Nemmers

10/27/14 2:22 PM

In our latest release, we've continued enabling IT organizations that want to provision and manage their applications more effectively. CloudBolt v4.6 makes providing self-service IT access to applications easier than ever, regardless of whether that application resides on a single server, or is a complete end-to-end stack of servers deployed across several environments. Like never before, users can interactively request entire stacks with just a few clicks, and deploy those stacks into any one of the dozen or so supported cloud and virtualization platforms.

We haven’t stopped there, though. In addition to streamlining the application provisioning process, we’ve put a significant amount of effort into other areas of CloudBolt as well. 

Nebula

We're also proud to announce an all-new connector for Nebula Private Cloud environments.

Image: CloudBolt now supports Nebula Private Cloud

Image: Add Nebula Resource Handler in CloudBolt Cloud Manager

With this new connector, CloudBolt customers gain the ability to deploy into and manage servers, applications, and even entire services in Nebula-backed environments. Nebula private cloud customers using CloudBolt gain immediate access to all of CloudBolt's features in both new and existing environments:

  • Chargeback and Showback
  • Reporting
  • Governance
  • Automated provisioning and management
  • Lifecycle management
  • Software license management
  • And more

Service Catalog

Customers are using the CloudBolt Service Catalog to provide end users self-service access to entire application stacks for some time now. In v4.6, we've updated the service creation process to make it even more straightforward. Just as they can do for the single server ordering process, admins can alter how the service ordering process looks for different end users and deployment environments. End users can be prompted to enter specific information as necessary based on their desired target deployment environment. 

Once ordered by a user, CloudBolt’s built-in approval mechanism can be leveraged for additional validation before CloudBolt steps through any number of automated processes required for delivery of a fully functional application stack.

The end result is clear: CloudBolt administrators can quickly create new service offerings that are able to span any supported target environment. Regardless of your platform of choice, CloudBolt can deliver a complete stack to your end users, and in less time than you think. 

Active Directory Group Mapping

Are you using one or more AD environments to authenticate CloudBolt users? In v4.6, admins gain the ability to map AD groups to CloudBolt groups. This AD group mapping also works with multiple AD environments, so if you're using CloudBolt in a multi-tenant capacity, you can still pick-and-choose how auth is handled for each tenant. 

Orchestration Hooks

Orchestration Hooks enable IT administrators to automate every step needed to deliver IT resources and applications to end-users. Extending on this capability, we've added a new Orchestration Hook type that enables the execution of an arbitrary remote script.

Image: Add a hook for remote script execution

This further extends CloudBolt’s lead as the most powerful cross-platform application deployment and management platform, as it can now be seamlessly integrated into nearly any manually-scripted provisioning and management process. Reusing your existing IP has never been easier or faster. 

Connector Improvements

Discovering and importing current state from existing environments is a CloudBolt Cloud Management key strength. In v4.6, this is even more thorough, as we now also detect all disk information from VMware vCenter virtual machines as well as AWS AMI, and Microsoft Azure public cloud instaces. 

Have a lot of VMs? Users in environments with tens of thousands of VMs will be happy to learn that VM discovery and sync is more efficient and faster. 

Puppet Enterprise users can now also leverage multiple Puppet environments rather than the default "Production". This can further help customers simplify their IT environments. 

Get It Now

The CloudBolt Cloud Manager v4.6 is available today via the CloudBolt support portal. Our updates are just another feature, and take mere minutes to complete

Don't have CloudBolt yet?

Schedule a Demo
or try it yourself
Read More

Topics: Public Cloud, VMware, Private Cloud, Upgrade, AWS, Puppet, azure, Nebula

Declare your Independence from Standard Cloud Management

Posted by Justin Nemmers

7/2/14 8:50 AM

Introducing the latest release of CloudBolt C2: v4.5

Connector Updates

With C2 v4.5, we’ve added two new connectors that further expand the breadth of technologies IT organizations can manage from a single-pane-of glass.

Google Compute Engine support gives administrators the ability to seamlessly offer end users controlled access to yet another public cloud provider. This includes the ability to install and manage applications from a supported configuration manager, as well as the ability to include GCE instances in C2 Service Catalog service templates.

Google Cloud Platform CloudBolt C2
Google Compute Engine CloudBolt C2
C2 v4.5 includes support for Google Compute Engine in the Google Cloud Platform.

We’ve also totally re-written and re-based our OpenStack connector. In this update, we’ve focused on compatibility, and we’re now able to support Icehouse, Havana, and Grizzly from the major OpenStack providers such as Mirantis. Of course, C2 can include OpenStack-backed resources when provisioning applications, running external flows, and accounting for licenses, just to name a few. C2 is already the best dashboard for OpenStack, and it’s getting even better with each release. No Horizon development needed!

openstack-cloud-software-vertical-large

We’ve also made some additional updates to our vCenter connector, including improved error handling when your VMware tools are out of date, and allowing for longer Windows hostnames. We’ve also made the Windows disk extending messages more clear and straightforward.

Amazon Web Services has also received some developer love. C2 now synchronizes both the public and private IP addresses for each AWS EC2 instance.

Configuration Management

We worked closely with the engineering team at Puppet, and now have a unique capability: C2 can now discover and import classes from a Puppet server.

Chef integration is even better: C2 now enables Chef bootstrapping on Windows and Ubuntu Linux systems.

User Interface Updates

Updates to the C2 UI are perhaps more subtle, but focused on helping users and administrators more effectively manage large numbers of applications and servers. We’ve integrated simple indicators describing the total number of selected items in each data table, making it much easier to manage large environments.

Did you know that you can use C2 to open network-less console connections on C2-managed servers? We’ve made this feature faster and more reliable in C2 v4.5.

Upgrading

Upgrading C2 is just like any other feature in C2: fast, easy, and predictable. Upgrading to C2 v4.5 is now even faster and easier than before.

Sounds Great, I Want It!

CloudBolt C2 has been recognized by Gartner for our industry-leading time to value. We effectively eliminate the barrier to entry for enterprise Cloud Management. C2 v4.5 is available today. Request a download, and you'll be up and running in your own environment in no time.

Schedule a Demo or try it yourself

Read More

Topics: Upgrade, AWS, Puppet, Chef, OpenStack, GCE

New Release: CloudBolt C2 v4.1 Cloud Manager

Posted by Justin Nemmers

9/26/13 3:22 PM


Customers are speaking, and we’re listening. C2 v4.1 contains a host of customer-suggested enhancements that are designed to enhance the capability and usability of C2’s management of a wide variety of IT resources.

CB Web Slides v4.1

Virtual Private Clouds and Amazon Web Services

In C2 4.1, the AWS connector now has the ability to detect, manage, and deploy systems into VPCs. C2 Environments backended by the AWS connector will now discover VPCs, VPC subnets, and other VPC-specific options such as security groups. C2 will also now auto-create C2 Environments for each selected VPC/Region, which nearly eliminates additional setup and configurations required.

VMware Integration Improvements

As we showed with our latest AWS updates, we’re always on the lookout for additional features to add to existing connectors. In this release, we’ve improved our VMware integration by adding more in-depth automatic C2 Environment creation for each VMware-hosted cluster present in vCenter. C2 will also now auto-discover any resource pool or datastore options as configured in vCenter, and will automatically create and map those options to C2 fields.

Re-Importing and Updating

Once AWS and VMware environments are created, C2 can now auto-detect and re-import changes to the underlying configuration of those environments. This includes things like VMware datastores, resource pools, and AWS security groups.

The Docs

We now include the full CloudBolt C2 product documentation as part of the install. The bottom of every page contains a link to access the embedded product documentation.

Speeds and Feeds

Despite our class-leading interface—recognized as the most intuitive available in a Cloud Manager, we’re always working to improve the user experience and keep it at the top. In C2 4.1, we’ve worked to get the rust out of many parts of the interface—so it will be faster than ever—especially on the server list page for environments that have thousands of servers. We’ve also polished the interface in several areas to help with usability. 

Additional Improvements

We already talk to Puppet Configuration Management, and now we speak that language even better with a updated Puppet connector in C2 4.1. 

We’re the only Cloud manager that can use itself to test your critical provisioning workflows. Configuring the Cloud Supply Chain Validator (CSCV) capability in C2 4.1 is now fully managed in the C2 UI, rather than requiring a configuration file modification.

For Red Hat Enterprise Virtualization Manager (RHEV-M), C2 will now automatically import the RHEV-M API certificate.

I want to see it!

Read More

Topics: VMware, Red Hat, Upgrade, Release Notes, AWS, vCenter, Puppet

CloudBolt Releases C2 v3.7.0

Posted by Justin Nemmers

4/8/13 9:03 AM

On behalf of the entire team here at CloudBolt, I’m excited to announce the release of CloudBolt C2 version 3.7.0. 

We continue to pull out all of the development stops in CloudBolt C2.  This latest version adds numerous improvements, and new features to help reduce the burden of IT and cloud management.

C2 updates in v3.7.0 make AWS easier

Amazon Web Services (AWS) support continues to strengthen.  Now, CloudBolt C2 will auto-create region-specific environments based on administrator-selected regions for EC2.  C2 will also list out the various AMIs admins want to make accessible to their users.  The best part is that this will work for any EC2-provided AMI, as well as customer-specific AMIs.  C2 also now provides for richer discover of running instances in EC2, so the server list and individual server views in C2 contain even more information about the related EC2 instance.  Keep your eyes peeled, because we’re going to continue adding capabilities to the AWS connector.

We believe that Network Virtualization from folks like Nicira by VMware will drastically change how IT organizations manage enterprises.  Our engineers have now enabled Network Virtualization support in the KVM connector, meaning administrators can now create and deploy virtualized networks on KVM-backed hosts using CloudBolt C2. 

One powerful aspect of CloudBolt C2 is that is it’s ability to apply actions to systems cross-environment.  Part of what was needed here was a multi-select capability that allowed users to multi-select where it makes sense to do so.  C2 now supports multi-select in the appropriate dialog boxes.

Provisioning instances is what CloudBolt C2 was built to do. When a user has no knowledge of what’s going on behind the scenes, it’s good practice to at least let them know that something is happening.  With 3.7.0, C2 does a better job showing users the provisioning progress of their instance.

Read More

Topics: Innovation, Feature, Cloud Manager, Upgrade, Release Notes, AWS

Build private cloud on top of virtualized network

Posted by Justin Nemmers

3/18/13 11:40 AM

Let’s face it. Networks are a pain to implement, maintain, and debug. Additionally, they’re often viewed as fragile enough that many teams generally wish to avoid routinely poking at them by messing with configurations or frequently creating/deleting VLANs.

Implementing a flexible and scalable private cloud environment on an inflexible network will only serve to reduce the flexibility and scalability of a private cloud environment that needs to grow.  In addition, ongoing management of these environments can quickly become difficult when administrators don’t have the ability to easily restrict network access by group, or have the ability to rapidly create new stand-alone networks for a specific application, group, or requirement.

virtualized networking separates logical from physical
Separate the logical from the phisical network.  Network virtualization does for networks what server virtualziation did for servers. You can't talk virtualization management without also talking about network virtualization management.

Enter network virtualization!  When implemented in your environment, and made consumable by a Cloud Manager, network virtualization suddenly breaks the network stack wide open.  In fact, I’d argue that until you virtualize the network, even private cloud alone is only partly useful.  Why?  Well, for several reasons:

  • Private clouds alone are limited by their ability to meet capacity demands. 
  • Eventually, that private cloud will run out of data center space, or will need to otherwise expand out of it’s shell. 
  • Whether your private cloud is fully on-prem, or you’re using a virtual private cloud model from someone like Amazon Web Services (AWS), the inflexibility of unifying that networking layer can be a difficult hurdle to surmount. 

Let’s expand on this AWS example.  Amazon offers a Virtual Private Cloud (VPC) that is essentially a private cloud hosted in the public cloud. Confused yet?  Don’t be. AWS uses advanced network and security parameters to effectively cordon off your cloud-based VMs from other tenants, allowing for secure communication and private networking in your hosted private cloud. They do this by manipulating the network layers in the hypervisors. AWS’ use of networking, although advanced, has its limitations, though. For instance, although VPCs can span availability zones, separate regions may require separate VPC definitions, leaving the networking integration to the user. In those cases, your local facility will have to implement it’s own routes to properly send traffic to the correct VPC. Although you can certainly work through those limitations, a hosted private cloud like that is wholly dependent on AWS. 

It doesn’t get any easier when your private cloud is completely on-prem. Be it demand growth, or a shift in requirements or priorities, networking is likely to be one of the significant bottlenecks in the growth and success of your private cloud.  

This is why a technology like network virtualization is so important. Implementing network virtualization in a private cloud environment (be it greenfield, or layered into an existing brownfield environment) allows you to approach new requirements with flexibility in mind and little concern over the networking infrastructure. Just make sure that your underlying network has the Layer 2 capacity for required traffic, and then start to build your environment above that.

In order to attain the flexibility of network virtualization on top of your private cloud, you need effective management. This goes beyond creating a handful of networks and handing them over to users.  Understanding what networks are required by which users and groups, and then ensuring that access is properly controlled is more than critical: it’s a requirement that must be met, or the network will remain a significant impedance to growth. Especially when it is time to expand the reach of your private cloud—whether that be adding capacity, layering in additional technologies, or perhaps looking to securely and safely make use of public cloud resources (congrats, you now have a hybrid cloud!)—Management of the entire stack is an imperative part of the solution. Deploy applications, resources, and networks all in one pass, no matter the environment. That’s the promise of network virtualization. CloudBolt makes it usable.

Read More

Topics: Network Virtualization, Software Defined Network, Management, Implementation, AWS

CloudBolt Releases C2 v3.6.1

Posted by Justin Nemmers

3/8/13 4:03 PM

We're keeping busy over here at CloudBolt.  To prove it, our engineers have released C2 v3.6.1 with a bounty of new capabilities.

Amazon Web Services (AWS for the acronym-inclined) support continues to get better and better in C2.  We understand that many users will want to import the existing state of an AWS account, so now C2 will import a list of all the running virtual instances in a customer's AWS accounts.  

Happy that you're able to provision servers quicker than ever?  Want to brag about it?  C2 now presents users that ability to post a Facebook message bragging about how well your IT environment is run after they provsion a virtual machine using CloudBolt. Of course, administrators can toggle that feature on and off as needed/required.

What happens when something goes wrong with a VM deployment? v3.6.1 now gives you more information on the status of an order, and will tell you more about errors from the underlying virtualization or configuration management.

While we'd love to say that bugs never happen, the fact is that all software has bugs.  We grabbed our fly swatter and fixed a handful of issues in this release, including UI rendering fixes for Internet Explorer, situations where the remote console feature failed, and some corrected verbiage in various status messages.

Ready to upgrade?  Hit up our support portal (login required) for details.  Want to kick the tires?  Request a download now!

Read More

Topics: Public Cloud, Feature, Cloud Manager, Upgrade, Release Notes, AWS, Development