/dev/zero - page 3
The infinite stream
-
Running a Text-based Kiosk with Systemd
Eucalyptus HQ has a big TV on the wall that displays the #eucalyptus-devel IRC channel so developers can always see what is going on and jump in if they need to. Until recently, a laptop drove that display, but that seemed like overkill to me, so I went to employ my Raspberry Pi running the Raspberry Pi Fedora Remix to do that instead. Since the IRC program it’s using, irssi, is text-based I don’t need to use any of the Pi’s precious little memory to run anything graphical, so I just needed to figure out how to make systemd spawn irssi instead of a login prompt on tty1.
I would normally do this by copying
/lib/systemd/system/getty@.service
to/etc/systemd/system/getty@tty1.service
and then editing that, but F18’s version of systemd let me do this in an even simpler manner. By creating a directory with the same name as that file, plus.d
, I can add a config file to that directory that overrides only the parts of the original unit file that I need to change:/etc/systemd/system/getty@tty1.service.d/kiosk.conf [Service] After=network-online.target Wants=network-online.target ExecStartPre=/usr/bin/nm-online ExecStart= ExecStart=/usr/bin/irssi KillSignal=SIGTERM StandardInput=tty StandardOutput=tty User=kiosk
Now I can just plug the system in and have it automatically up and running irssi in less than a minute.
Unexpected lessons
I didn’t expect to have to run
nm-online
here becausenetwork-online.target
is supposed to wait for a service that runs that itself, but for some reason systemd didn’t order things that way and irssi came up before the network connection did. Running that command as part of this unit worked around that problem.Use the
consoleblank=0
kernel parameter to prevent Linux from blanking the screen after the usual ten minutes of inactivity.I’m using the TV’s USB “service” port to power the raspberry pi. That usually works just fine, but when the TV turns off it cuts the power to that port as well, abruptly shutting the raspberry pi off. I don’t have any data loss in particular to worry about, but turning the system back on causes some annoyance: when the TV turns on the raspberry pi also powers on and attempts to detect what kind of screen it is plugged into. At that point the TV hasn’t figured out what it wants to display yet, so the detection fails and I’m left with a blank screen until I reboot the computer.
-
Managing a DNS Domain from One Place
Taking a DNS name and resolving it to the address of a machine is easy to understand and easy to implement if you’re an administrator. Doing a reverse lookup from an address back to a name, however, is more difficult due to the way addresses are divided up. I won’t attempt to describe the details here (I recommend Liu and Albitz’s DNS and BIND for the gory details), but in short, the way this works is by breaking an IP address into its four octets and handling them from there like regular hierarchical names in the special
in-addr.arpa
zone:1.2.0.192.in-addr-arpa. PTR foo.example.com.
This is problematic for two main reasons:
- You have to change two zones every time you change a DNS name.
- If you have fewer than 256 addresses, your ISP can’t delegate the appropriate subset of the
in-addr.arpa
zone to you so you can maintain it yourself. This usually forces you to log into a web page provided by your ISP every time you change a DNS name.
RFC 2317 notes that you can work around this by filling up your subset of the
in-addr.arpa
zone withCNAME
records instead of the usualPTR
records like this:$ORIGIN 2.0.192.in-addr.arpa. 1 CNAME 1.ip4.example.com. 2 CNAME 2.ip4.example.com. ... 253 CNAME 253.ip4.example.com. 254 CNAME 254.ip4.example.com.
After you set this up you can control your forward and reverse DNS records from the same place without needing to change the reverse zone you just set up:
$ORIGIN example.com. foo A 192.0.2.1 bar A 192.0.2.2 ... baz A 192.0.2.253 bop A 192.0.2.254 1.ip4 PTR foo 2.ip4 PTR bar ... 253.ip4 PTR baz 254.ip4 PTR bop
Of course, if you rely on your ISP to create reverse DNS names for you they have to be willing to create non-
PTR
records before you can take advantage of this.If you’re lucky enough to have an entire /24 block of addresses all to yourself you can simplify the reverse DNS zone by simply mapping the entire set of addresses with a single
DNAME
instead of a long list ofCNAME
s:$ORIGIN 2.0.192.in-addr.arpa. @ DNAME ip4.example.com.
This has the same net effect as the list of CNAMEs, but it shortens things significantly.
-
What's New in Euca2ools 3, Part 2: A Developer's Perspective
The upcoming version of euca2ools, version 3, completely reworks the command line suite to make it both easier to write and easier to use. Part 1 of this series discussed the user-facing changes version 3 has to offer, and today we’re going to take a look at how things improve on the developer’s side of the fence.
A change in philosophy: declarative programming
The developer is very much in the driver’s seat in version 1 of euca2ools. To use a car analogy, the developer directly controls the code’s direction, speed, and gearbox manually. Version 2 adds a cruise control by centralizing a lot of boilerplate code in the form of boto’s
roboto
module. Version 3 opts to let the developer give the requestbuilder framework a destination, step aside completely, and let it do the driving for the boring parts of the trip.Requestbuilder offers a set of base classes and a domain-specific language based on python’s standard
argparse
library that allows the developer to say exactly how something should look at the command line in addition to how it should look when given to the server all in the same place.What makes this so powerful is that it lets anybody with a service’s documentation and knowledge of how to use
argparse
write a command line tool quickly and painlessly. For instance, it took me around a day to write highly-customized command line tools for every operation Amazon’s Elastic Load Balancing service supports. Here’s the code from one of them:class CreateLBCookieStickinessPolicy(ELBRequest): DESCRIPTION = ('Create a new stickiness policy for a load balancer, ' 'whereby the load balancer automatically generates cookies ' 'that it uses to route requests from each user to the same ' 'back end instance. This type of policy can only be ' 'associated with HTTP or HTTPS listeners.') ARGS = [Arg('LoadBalancerName', metavar='ELB', help='name of the load balancer to modify (required)'), Arg('-e', '--expiration-period', dest='CookieExpirationPeriod', metavar='SECONDS', type=int, required=True, help='''time period after which cookies should be considered stale (default: user's session length) (required)'''), Arg('-p', '--policy-name', dest='PolicyName', metavar='POLICY', required=True, help='name of the new policy (required)')]
The framework hands everything inside each
Arg
in this code toargparse
to gather input from the command line and then send the results directly to the web server using whatever name argparse gives the input it gets. For instance, whatever a user supplies using the-e
option ends up getting sent to the server as aCookieExpirationPeriod
parameter. With a small amount of practice it becomes quite easy to write a bunch of commands this way very quickly.One request, one command
Euca2ools are built around a “one request, one command” tenet. This means that, in general, there is a dedicated command for each thing a web service can do. This philosophy naturally lends itself to the tight coupling between command line options and what gets sent to the server discussed earlier, but it also lends itself to reversing the usual relationship between web services and web service requests. Whereas one typically writes an object that represents the service and uses methods on it to send requests, in euca2ools it is the commands, and thus the requests, which are the first-class citizens. Each command that represents a request instead points to a service, rather than the other way around.
The way this works in practice is by defining a base class for each service and a base class that all methods which use that service share:
class CloudWatch(requestbuilder.service.BaseService): NAME = 'monitoring' DESCRIPTION = 'Instance monitoring service' API_VERSION = '2010-08-01' AUTH_CLASS = requestbuilder.auth.QuerySigV2Auth URL_ENVVAR = 'AWS_CLOUDWATCH_URL' ARGS = [MutuallyExclusiveArgList( Arg('--region', dest='userregion', metavar='USER@REGION', route_to=SERVICE, help='''name of the region and/or user in config files to use to connect to the service'''), Arg('-U', '--url', metavar='URL', route_to=SERVICE, help='instance monitoring service endpoint URL'))] class CloudWatchRequest(requestbuilder.request.AWSQueryRequest): SERVICE_CLASS = CloudWatch
Services can supply their own command line options in the same way as requests. After it gathers options from the command line, requestbuilder uses
route_to
to choose where to send it. This also provides a convenient way to tell the framework not to send an option to the server at all when a command needs to process it specially: just useroute_to=None
.Convention over configuration
The oft-quoted programming paradigm for frameworks is just as true for euca2ools 3 as it is elsewhere. Want to make a command print something? Just write a
print_result
method. The result from the server gets passed in as a dictionary.class TerminateInstances(EucalyptusRequest): DESCRIPTION = 'Terminate one or more instances' ARGS = [Arg('InstanceId', metavar='INSTANCE', nargs='+', help='ID(s) of the instance(s) to terminate')] LIST_TAGS = ['instancesSet'] def print_result(self, result): for instance in result.get('instancesSet', []): print self.tabify(('INSTANCE', instance.get('instanceId'), instance.get('previousState', {}).get('name'), instance.get('currentState', {}).get('name')))
Want to make a request do fancier preparations than
argparse
can do on its own? Just write apreprocess
method that takes things fromself.args
and adds things toself.params
to be sent to the server.class DescribeSecurityGroups(EucalyptusRequest): DESCRIPTION = ('Show information about security groups\n\nNote that ' 'filters are matched on literal strings only, so ' '"--filter ip-permission.from-port=22" will *not* match a ' 'group with a port range of 20 to 30.') ARGS = [Arg('group', metavar='GROUP', nargs='*', route_to=None, default=[], help='limit results to specific security groups')] ... def preprocess(self): for group in self.args['group']: if group.startswith('sg-'): self.params.setdefault('GroupId', []) self.params['GroupId'].append(group) else: self.params.setdefault('GroupName', []) self.params['GroupName'].append(group)
There are also a few other methods one can plug in, such as
postprocess
, and, for especially early-running code,configure
. Expect documentation for requestbuilder that covers this in detail in the future.Scratching the surface
The examples above cover only a fraction of what is possible with euca2ools 3’s new infrastructure. While you can look forward to some more advanced uses of it in later blog posts, you can also take a look at the current euca2ools code in development to see some of the interesting things one can do with it. Today’s pre-release of that code carries with it commands for all three of AWS’s “triangle” services: Auto Scaling, CloudWatch, and Elastic Load Balancing. Continuing what seems to have become a euca2ools tradition, just look for the commands that start with
euscale
(pronounced “you scale”)euwatch
(“you watch”), andeulb
(“you’ll be”).Packages for Fedora and RHEL 6 are available here. If you’re using another OS or want to build the code yourself you can simply clone euca2ools’s git repository’s
requestbuilder
branch. Requestbuilder itself is available on PyPI and GitHub. As always, I encourage you to test this code against AWS and Eucalyptus 3.3 and let me know what you think on the euca-users mailing list. If you encounter bugs, please file them in the project’s bug tracker.
-
What's New in Euca2ools 3, Part 1: A User's Perspective
Version 3 of euca2ools, slated for release in just a couple months, gives the command line suite a much-needed refresh that makes it both easier to write and easier to use. Most of the innovation here involves changes to the platform upon which it is built. I will cover those changes from a developer’s perspective in future blog posts, but today I’m going to focus on what euca2ools 3 brings to the table for developers and other users alike. While there are too many small improvements to possibly cover them all, euca2ools 3 at last brings a few of the niceties power users have come to expect from their command line tools to cloud management.
A configuration file
Yes, you read that right: a configuration file. Both euca2ools and the command line tools provided by AWS themselves have astonishingly limited support for configuration, forcing people to resort to writing a separate shell script for each combination of users and clouds one might possibly want to access and then use them in place of one.
Your cries of anguish have been heard, so now we have this:
[user gholms] key-id = AKIA93F29V0AEXAMPLE secret-key = vcasd93cm1458un4vj84039vda78mDEXAMPLE [user ecc-admin] key-id = EVDB93F29V0AEXAMPLE secret-key = 38fva93cm1458un4vj84039vda78mDEXAMPLE [region us-east-1] ec2-url = https://ec2.amazonaws.com/ iam-url = https://iam.amazonaws.com/ s3-url = https://s3.amazonaws.com/ user = gholms [region ecc] ec2-url = https://communitycloud.eucalyptus.com:8773/services/Eucalyptus/ iam-url = https://communitycloud.eucalyptus.com:8773/services/Euare/ s3-url = https://communitycloud.eucalyptus.com:8773/services/Walrus/ user = ecc-admin [global] default-region = us-east-1
A file like this, combined with the
--region
option that all tools share, mean you can mix and match users and clouds to you heart’s content. Just throw a file like this inside of~/.euca
, end it with.ini
, and away you go! You can add as many files to~/.euca
as you want — they all get combined together.Friendly error feedback
Another common complaint that people had with euca2ools 2 was its behavior in the face of input that didn’t match what it expected. Some of the worst offenders had error messages ranging from confusing to irrelevant to nonexistent. Euca2ools 3 overhauls the code that does this, replacing it with standard python tools and friendlier code that makes its behavior in the face of errors much better.
Here’s how it behaves in the face of the most common case of this:
$ euca-describe-availability-zones error: missing access key ID; please supply one with -I
Also included is special treatment for “pick one from multiple alternatives” options:
$ euare-useraddcert usage: euare-useraddcert (-c CERT | -f FILE) [-u USER] [--as-account ACCOUNT] [--region REGION | -U URL] [-I KEY_ID] [-S KEY] euare-useraddcert: error: one of the arguments -c/--certificate-body -f/--certificate file is required
A lot of attention to detail went into dealing with some of the most common mistakes people make:
$ euca-register -n myimage -b /dev/sda1=snap-12345678:false euca-register: error: argument -b/--block-device-mapping: second element of EBS block device mapping "/dev/sda1=snap-00000000:false" must be an integer $ euca-authorize mygroup -p 8773:8777 euca-authorize: error: argument -p/--port-range: multi-port range must be separated by "-", not ":"
Tagging and filtering support
Euca2ools 3 at last offers full support for EC2’s massive sets of resource tags and filters:
$ euca-describe-instances -h usage: euca-describe-instances [-h] [--show-empty-fields] ... --filter NAME=VALUE restrict results to those that meet criteria ... allowed filter names: architecture CPU architecture availability-zone block-device-mapping.attach-time volume attachment time block-device-mapping.delete-on-termination whether a volume is deleted upon instance termination block-device-mapping.device-name volume device name (e.g. /dev/sdf) block-device-mapping.status volume status block-device-mapping.volume-id volume ID client-token idempotency token provided at instance run time dns-name public DNS name group-id security group membership hypervisor hypervisor type image-id machine image ID instance-id instance-lifecycle whether this is a spot instance instance-state-code numeric code identifying instance state instance-state-name instance state instance-type ip-address public IP address kernel-id kernel image ID key-name key pair name provided at instance launch time launch-index launch index within a reservation launch-time instance launch time monitoring-state whether monitoring is enabled owner-id instance owner's account ID placement-group-name platform whether this is a Windows instance private-dns-name private-ip-address product-code ramdisk-id ramdisk image ID reason reason for the more recent state change requestor-id ID of the entity that launched an instance reservation-id root-device-name root device name (e.g. /dev/sda1) root-device-type root device type (ebs or instance-store) spot-instance-request-id state-reason-code reason code for the most recent state change state-reason-message message for the most recent state change subnet-id ID of the VPC subnet the instance is in tag-key name of any tag assigned to the instance tag-value value of any tag assigned to the instance tag:KEY specific tag key/value combination virtualization-type vpc-id ID of the VPC the instance is in
The new foundation this code is based upon makes it incredibly simple to extend support for these features as things change in the future.
What else?
Some other minor, but nonetheless noteworthy, changes include:
euca-*
tools gained a--show-empty-fields
option that tweaks their output to make it friendlier for running through thecolumn
command.- All tools that access web services use the same options (
-I
and-S
) for access keys. euare-*
tools’--delegate
option for cloud administrators is now--as-account
.- Multiple
--filter
options are handled correctly. - Machine image device mappings are now handled correctly.
A few tools have yet to be ported to the new framework, but will be in the near future.
eustore-installimage
is known to be broken. The bundle management tools should work correctly, though their testing to date has been minimal. Finally, do not install them on a system that runs a Eucalyptus node controller.Isn’t aws-cli the future? Why continue developing euca2ools?
Aws-cli is a great project. Both it and euca2ools tie what the server sees very closely to what the user sees under the hood, but the euca2ools suite does so in a way that makes it trivial to customize tools to do more complicated things behind the hood or to make them easier to use. For instance, consider changing a security group’s permissions in EC2 with aws-cli:
$ aws ec2 authorize-security-group-ingress --group-name MySecurityGroup --ip-permissions '{"from_port":22,"to_port":22,"ip_protocol":"tcp","ip_ranges":["0.0.0.0/0"]}'
The exact format we need to use to supply the info the tool needs requires relatively detailed knowledge of what EC2-the-server expects. Compared to that, the euca2ools version of that is easier to remember and much easier to type:
$ euca-authorize MySecurityGroup --port 22 --source-subnet 0.0.0.0/0
Aws-cli is a very young project, so people haven’t yet had the chance to iron it out completely. Perhaps some day it will become as user-friendly as euca2ools and finally eclipse it. But we aren’t there yet.
How can I try it out?
If you’re interested in a preview of the next major version of euca2ools, an alpha release is available on GitHub. In addition to the dependencies required to run euca2ools 2, you will also need to install requests and the new requestbuilder framework that drives the new tools. It is still alpha-quality software, so be prepared to find bugs. If you encounter any, feel free to file them in the euca2ools project’s bug tracker.
If you’re interested in helping with development, we are happy to accept pull requests on GitHub. Please also consider joining the euca-users mailing list or stopping by in the
#eucalyptus-devel
IRC channel on Freenode. I look forward to hearing your feedback. 8^)
-
How to Override a Class Method in Python
A class method in python differs from an instance method in a couple important ways:
- It binds to a class rather than an instance (hence its name). Thus, its first argument is a class, often called
cls
rather than the usualself
. - It can be called on both an instance of a class and the class itself.
In general, they behave similarly, but one area in which they can differ is when we go to override the class method:
class Spam(object): @classmethod def parrot(cls, message): print cls.__name__, "says:", message class Eggs(Spam): @classmethod def parrot(cls, message): Spam.parrot(cls, message)
This code is broken because
Spam.parrot
is already bound to theSpam
class. This meansSpam.parrot
’scls
argument will be theSpam
class rather than theEggs
class that we wanted, soSpam.parrot
will end up being called in the wrong context. Even worse, everythingEggs.parrot
passed to it, includingcls
, ends up getting passed as regular arguments, resulting in disaster.>>>> Spam.parrot("Hello, world!") Spam says: Hello, world! >>>> Eggs.parrot("Hello, world!") Traceback (most recent call last): File "<console>", line 1, in <module> File "<console>", line 4, in parrot TypeError: parrot() takes exactly 2 arguments (3 given)
To chain up to the
Spam
class’s implementation we need to use asuper
object, which will delegate things the way we want.class Eggs(Spam): @classmethod def parrot(cls, message): super(Eggs, cls).parrot(message)
There’s a shortcut for the last line if you’re using python 3:
super().parrot(message)
The
super
object functions as a proxy that delegates method calls to a class higher up in theEggs
class’s hierarchy, and in this case it is critical in ensuring that it gets called in the right context.>>>> Spam.parrot("Hello, world!") Spam says: Hello, world! >>>> Eggs.parrot("Hello, world!") Eggs says: Hello, world!
If you haven’t used
super()
before, here’s what it’s doing:- The python interpreter looks for
Eggs
incls.__mro__
, a tuple that represents the order in which the interpreter tries to match method and attribute names to classes. - The interpreter checks the class dictionary for the next class that follows
Eggs
in that list that contains"parrot"
. - The
super
object returns that version of"parrot"
, bound tocls
, using the attribute-fetching__get__(cls)
method. - When
Eggs.parrot
calls this bound method,cls
gets passed toSpam.parrot
in place of theSpam
class.
In general I tend to stick with the older-style syntax for chaining method calls, but this is one case where
super()
is simply indispensable.
- It binds to a class rather than an instance (hence its name). Thus, its first argument is a class, often called