mbox.search(None, 'UNSEEN SINCE 1-Jan-2010 NOT FROM "Smith"')

Like many other IMAP4 object methods, you won’t find the options for this search parameter in the Python imaplib documentation.  This is because those parameters are specified in detail inside RFC3501; The Internet Message Access Protocol – Version 4rev1 standard.

RFC’s are great; they’re very detailed, but they were never designed to be user-friendly how-to’s.  For example the search string parameter options accepted by IMAP4′s  SEARCH command are strewn across five (5) different pages in the RFC and are listed in alphabetical order rather than grouped by function.

For future reference I decided to create an IMAPv4 SEARCH command reference, i.e. cheat-sheet. I’ve grouped the commands by:

• Search for String in Message commands
• Search for Message with Flags Set commands
• Search for Messages with Flags Not Set commands
• Search on Internal Message Date commands
• Search on Message Header Date commands
• Search on Message Size commands

Note: If you use Lotus Notes the mailbox must be full-text indexed before the IMAP4 SEARCH command will work.

The examples below use:

• “Routers” as the Rancid group-name of interest. You can replace it with whatever group you want to look at.
• /var/lib/rancid as the Rancid home directory, default for the Ubuntu Rancid package. If you are using a different Linux distro, just replace it with the correct home directory.

CVS checkout:

First you need to checkout a local copy, by default it will go in the current directory.
sudo cvs -d /var/lib/rancid/CVS co Routers

To get status:

sudo cvs -d /var/lib/rancid/CVS status Routers

To get a revision history:

sudo cvs -d /var/lib/rancid/CVS rlog Routers

To compare versions:

sudo cvs -d /var/lib/rancid/CVS diff -r 1.2 -r 1.3 Routers

To replace a file:

Delete the file and use the following command …
sudo cvs -d /var/lib/rancid/CVS update Routers

I use a wide variety of Unix text processing tools on a regular basis, but over time, like many others I started migrating those tasks that required the power of ‘awk’ over to another language; in my case that language was Python.  Typically I can write scripts faster in Python and I find that the code is more readable. However, after reading the above post I was reminded that there are some one-liner ‘awk’ tasks that are really clean and effective. Lately I have found myself starting to sparingly use ‘awk’ again, here’s why…

When to use ‘awk’ instead of ‘cut’

1. Cut’s delimiter is a single character, awk’s delimiter is a regular expression.
2. Awk allows fields to specified relative to the last field position using ‘NF’.
3. Cut always displays fields in order of ascending field number, regardless of the order fields are specified in the field list parameter, awk can redisplay the fields in any order that you specify.

Examples:

splits fields at multiple characters either a, b, c, d
awk -F'[abcd]'

split at one (1) or more spaces
awk -F' +'

re-order fields
awk '{print $3 "\t" $2 "\t" $1}'

prints last field
awk '{print $NF}'

prints next to last field
awk '{print $(NF-1)}'

When to use ‘awk’ instead of Python, Perl, etc.?

1. When you can write the task in one simple, readable line with awk, i.e.
   1. Simple reformatting of data.
   2. Simple comparisons on fields.
   3. Rearrange order of fields.
   4. Split on regular expressions, including multiple characters.
   5. Feel free to comment on other reasons.
2. When the speed of Python, Perl, etc. scripts are too slow for repeated use, this is rare when coded properly.

Watch your quotes with ‘awk’ …

Here is the standard unix method of quoting:

$ awk '$NF > 385 && $(NF-1) ~ "^Sh" {print NR "\t" $0}' orders.txt
4       10416   2005-05-10 00:00:00     Shipped 386
6       10418   2005-05-16 00:00:00     Shipped 412

Here is the equivalent command using unxutils for Windows:

Note the difference in quoting…

C:\> gawk "$NF>385 && $(NF-1) ~ \"^Sh\" {print NR \"\t\" $0}" orders.txt
3       10416   2005-05-10 00:00:00     Shipped 386
5       10418   2005-05-16 00:00:00     Shipped 412

Here is what the above command is doing:

1. Iterates through every line in the file “orders.txt”.
2. Splits the fields at tab characters (default delimiter).
3. Tests if the last field is greater than 385.
4. Tests if the next to last field matches the regular expression “^Sh”, i.e. begins with the letters “Sh”.
5. If items 3 & 4 were true then print the line number followed by a tab and then then line text itself.

Sample text being processed:

$ cat orders.txt
10413   2005-05-05 00:00:00     Shipped 175
10414   2005-05-06 00:00:00     On Hold 362
10415   2005-05-09 00:00:00     Disputed        471
10416   2005-05-10 00:00:00     Shipped 386
10417   2005-05-13 00:00:00     Disputed        141
10418   2005-05-16 00:00:00     Shipped 412
10419   2005-05-17 00:00:00     Shipped 382
10420   2005-05-29 00:00:00     In Process      282

In the electrical engineering world we used lots of schematics; schematics didn’t waste valuable real-estate with actual pictures of the resistors, capacitors, inductors, diodes, transformers, etc.   Why?  Because it was supposed to be a design & troubleshooting tool, not a flipp’n art & crafts project!

As an experiment try using geometric shapes, i.e. triangles, pentagons, octagons, hexagons, etc. to represent network devices.

1. Use a geometric shape that matches the number of interconnecting interfaces, then label each  inside  corner of the shape with interface specific info.
2. Use the remaining interior of the shape to record device specific information, i.e. hostname, IP/mask, serial number, firmware/software version, etc.
3. Fill in geometric shapes with a color to identify the device type, i.e. red for firewalls, orange for IPS’s, blue for routers, green for switches, etc.

Whether or not you like the idea of using geometric shapes in your network drawings, at least consider putting the emphasis on creating useful schematics for design & troubleshooting; the quality and presentation of the data and the interconnects should always trump the artwork.   Chances are anyone else using your drawings will also appreciate it.

Thanks to J. Scott Haugdahl’s for his book Network Analysis and Troubleshooting, it was from his book that I first got the idea of using geometric shapes in network drawings.

• https://github.com/bpeterso2000/pycobol
• https://github.com/bpeterso2000/Harbingers-Hollow

You can pipe the results of copybook2csv.py into this module to quickly parse the data into a list, or once you already know the structure you can call parse_data(struct_fmt_string) directly. If the copybook field and actual record lengths don’t match it will still parse the data, but it will display a warning indicating that the data could be truncated or needed to be padded to fit the field definitions.

#!/usr/bin/env python
# -*- coding: utf-8 -*-
__version__ = """COBOL Fixed-length Data Parser ver 0.2
Note: This version does not work with OCCURS in Copybook files,
but is a lot faster than the varaible length data parser modules.

License: GPLv3, Copyright (C) 2010 Brian Peterson
This is free software.  There is NO warranty; 
not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
"""
USAGE = """copybook2list.py CopybookFile"""

import load
import csv, struct, sys

def parse_data(struct_fmt, lines):
    try:
      return [ struct.unpack(struct_fmt, i) for i in lines ]
    except struct.error:
        sys.stderr.write('Record layout vs. record size mismatch\n')
        size = sum([ int(i) for i in struct_fmt.split('s')[:-1] ])
        return [ struct.unpack(struct_fmt, i.ljust(size)[:size]) 
          for i in lines ]

def main(args):  
    copybook = load.csv_(args.copybook.readlines(), strip_=True)[1:]
    field_lengths = [ int(i[2]) for i in copybook ]
    struct_fmt = 's'.join([ str(i) for i in field_lengths ]) + 's'
    if args.struct:
        print struct_fmt
    else:
        for record in parse_data(struct_fmt, load.lines(args.datafile)):
            print record

if __name__ == '__main__':
    from cmd_line_args import Args
    args = Args(USAGE, __version__)
    args.allow_stdin()
    args.add_files('datafile', 'copybook')
    args.parser.add_argument('-s', '--struct', action='store_true',
        help='show structure format')
    main(args.parse())

• Ability to display multi-line version information strings (argparse ignores line feeds), used with the --version option.
• Ability to include positional filename arguments with very little code.
• Ability to allow the last positional filename argument to be replaced with Stdin.
• Ability to call common options from a standard command-argument library.

Examples:

#!/usr/bin/python
__version__ = """argparse wrapper sample code version 0.1a

This is free software.  There is NO warranty; 
not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
"""

USAGE = """cobol2csv.py [OPTIONS] COPYBOOK [DATAFILE]
COPYBOOK - Filename: output from copybook2csv.py
DATAFILE - Filename: COBOL records, fixed-width text
"""

import load

def main(args):
    fields = load.csv_(args.copybook, strip_="right", prune=True)
    data = load.lines(args.datafile, stop_at_line=1)
    if args.verbose:
        print 'In verbose mode...'
    if args.license:
        print 'GPL (GNU Public License)'

if __name__ == '__main__':
    from cmd_line_args import Args
    args = Args(USAGE, __version__)
    args.allow_stdin()
    args.add_files('copybook', 'datafile')
    args.add_options('debug', 'verbose')
    args.parser.add_argument('--license', action='store_true', 
        help='display license information')
    main(args.parse())

Explanation:

• Adds 2 positional filename parameters: copybook & datafile
• The ‘datafile’ parameter can be optionally omitted and Standard Input (Stdin) can be used instead of reading from a file. For example cat file1.txt|./cobol2csv.py layout.csv will work.
• -d, --debug, -v, --verbose are automatically added from a standard library of command-line options
• The argparse object can be accessed directly to support all normal argparse methods & functionallity (i.e. see --license option).

Note: The source code below may be dated, to get the most current version visit Harbingers-Hollow at Github.

import argparse, sys

__all__ = ['Args']

class VersionAction(argparse.Action):
    """Overrides argparse_VersionAction(Action) to allow line feeds within
    version display information.""" 
    def __init__(self, option_strings, version=None, 
         dest=None, default=None, help=None):
         super(VersionAction, self).__init__(option_strings=option_strings,
             dest=dest, default=default, nargs=0, help=help)
         self.version = version

    def __call__(self, parser, namespace, values, option_string=None):
        version = self.version
        if version is None:
            version = parser.version
        print version
        parser.exit()

class Args:
    """argparse wrapper"""
    
    allow_stdin = False
    
    def __init__(self, usage, version):
        self.parser = argparse.ArgumentParser(usage=usage)
        self.parser.version = version
        self.parser.add_argument('-V', '--version', 
            action = VersionAction,
            help='display version information and exit')
    
    def add_files(self, *file_args):
        """Add positional filename argurments.  If self.allow_stdin is set
        Example:
            object.add_filenames('config_file', 'data_file'])
            The 1st filename will be saved in a variable called 'config_file'.
            The 2st filename will be saved in a variable called 'data_file'.
        """
        if self.allow_stdin:
            for file_arg in file_args[:-1]:
                self.parser.add_argument(file_arg, help='filename... %s' % file_arg)
            self.parser.add_argument(file_args[-1], 
                help='filename... %s' % file_args[-1], nargs='?')
        else:
            for file_arg in file_args:
                self.parser.add_argument(file_arg, help='filename... %s' % file_arg)      
        self.file_args = file_args
  
    def add_filelist(self):
        """FUTURE: ability to add a list of files to be processed.  Similar to
        python filelist module, but with ability to include other arguments.
        Support for wildcards."""
        pass
        
    def add_options(self, *options):
        """Add from a standard library of pre-defined command-line arguments"""
        for option in options:
            option = option.lower()
            if option == 'debug':
                self.parser.add_argument('-d', '--debug', action='store_true',
                    help='Turn on debug mode.')
            elif option == 'debug_level':
                self.parser.add_argument('-d', '--debug', type=int,
                    help='Set debug level 1-10.')
            elif option == 'verbose':
                self.parser.add_argument('-v', '--verbose', action='store_true',
                    help='Turn on verbose mode.')
            elif option == 'quiet':
                self.parser.add_argument('-q', '--quiet', action='store_true',
                    help='Suppress all output to terminal.')
    
    def allow_stdin(self):
        self.allow_stdin = True

    def parse(self):
        """Parse args & use sys.stdin if applicable
        Sets all file arguments to a file read object"""
        args = self.parser.parse_args()
        if self.file_args:
            if self.allow_stdin:
                if not sys.stdin.isatty():
                    setattr(args, self.file_args[-1], sys.stdin)
                else:
                    self.allow_stdin = False
            last_arg_idx = len(self.file_args) - self.allow_stdin
            for file_arg in self.file_args[:last_arg_idx]:
                try:
                    file_ = open(getattr(args, file_arg))
                except IOError, error_msg:
                    sys.stderr.write('ERROR loading file "%s".\n%s\n' % 
                        (file_arg, error_msg))
                    sys.exit(1)
                setattr(args, file_arg, file_)        
        return args   

Configure SMTP server settings

• Select Administration –> General Setup
• Enter parameters under SMTP (Outgoing Email) Settings. For more information visit Eventum Outgoing Email Setup

Process outgoing e-mail

Eventum outgoing e-mail will queue up until the process_mail_queue.php script is run. Setup a cron job for this, i.e.

* * * * * /usr/bin/php /var/www/eventum/misc/process_mail_queue.php

Incoming Mail Setup

Setup email account for new tickets

Setup an account on your email server where new tickets can be sent, i.e. “Eventum Helpdesk” eventum@<domain_name>

Setup the email account in Eventum to download new tickets

Select Administration –> Manage Email Accounts
Create the account, i.e. Associated Project: “Incident Tracking”, Hostname: “Eventum Helpdesk”, Type: pop3, Port: 110, Auto-Creation of Issues: Enabled.

Enable Eventum e-mail integration

• Select Administration –> General Settings
• Select Yes under Allow Un-Assigned Issues?
• Select Enabled under Email Integration Feature. For more information visit Eventum Email Integration Setup

Setup a cron job to download the new ticket emails

* * * * * /usr/bin/php -f /var/www/eventum/misc/download_emails.php "Eventum Helpdesk" <email_server_hostname>

Setup email account for association with existing tickets (subject-based)

Setup an account on your email server where email can be sent that is associated with an existing ticket, i.e. “Eventum Ticket” eventum.ticket@<domain_name>

Enable Eventum subject-based e-mail association

• Select Administration –> General Settings
• Select Enabled under Subject Based Routing.

Setup the email account in Eventum to download associated tickets

Select Administration –> Manage Email Accounts
Create the account, i.e. Associated Project: “Incident Tracking”, Hostname: “Eventum Ticket”, Type: pop3, Port: 110, Auto-Creation of Issues: Disabled.

Setup a cron job to download emails associated with tickets

* * * * * /usr/bin/php -f /var/www/eventum/misc/download_emails.php "Eventum Ticket" <email_server_hostname>

Additional Eventum E-mail Integration Documentation

See Email Routing Interface for information on:

• Setting up email routing with Sendmail
• Setting up email routing with qmail
• Setting up email routing with postfix
• Setting up email routing with exim
• Setting up email routing with 1 email account for multiple projects

Special thanks to Jeremy Dye for contributions to this post.

In the war room

After mounting a large flat screen LCD TV to display real-time Opsview’s viewport grids & Syslog-ng Log-Reaver translations we came across an interesting challenge – Opsview page headers, menus & footers gobbled up about half of our viewport screen real-estate, as a result the viewport grid didn’t fit on the TV without scrolling. At first we looked into possibly of tossing together some custom code to display stripped down viewports, but we didn’t get very far before Scott Burningham came up with a brilliant idea – why not use Chrome’s AdBlock extension to hide all of Opsview’s header, menu and footer div elements? This solution totally cleaned up the viewport and it works like a charm.

Opsview viewport grids

Viewports in Opsview allow us to assign keywords to any combination of monitored services & hosts and then generate labeled cells that represent aggregated views of those statuses. This allows us to display statuses for important business processes/services & applications rather than just individual pieces of I.T. equipment and their hosted-services.

I have been really happy with Opsview’s viewport grid (cells) option in the 3.7.x release. In previous releases of Opsview the viewport cells only allowed the user to list items vertically and it didn’t take very many items before they would fall off the screen. By combining the viewport grid option with the Chrome AdBlock extension we can display a large array of highly-visible status lights.

Opsview viewports work extremely well once you understand them, but out of the box it’s not particularly intuitive as to how the contact’s authorized keywords configuration settings affect viewport displays. In each contact configuration page there is a section with checkboxes for configuring the user’s authorized keywords, these authorized keywords determine which items/cells will show up by default in the viewport page.  You can manually set the viewport filter to override these defaults and display unauthorized keyword cells, however, each time you leave and return to the viewport page it will revert to hiding all viewport cells not associated with an authorized keyword.

Here is a bare-bones mach-up, the actual production application includes scores of service specific applications that I can’t show here, but with a little imagination you can envision what it looks populated with all those extra cells nicely represented in one dashboard view.

Syslog-ng Log-Reaver translations

A really cool log tool, I will be releasing the details of this free open source application in future posts… stay tuned.

\033[x;ymTEXT\033[0m

ANSI Color Pairs (x;y)

X values (attribute/intensity)

Y values (color)

Additional attributes & background colors

You may read about ANSI codes that support strike-through, blinking, underline, dimming, background colors, etc. Those are not universally supported and their results may vary on different ANSI terminals. The attributes & colors I listed above should work with predictable results on any ANSI terminal.

Setting ANSI colors using echo

To set a red color prompt…

echo "^[[7;31mCRITICAL^[[0m"

Setting ANSI colors using sed

sed ''/crit/s//$(printf "\033[31mCRITICAL\033[0m")/g''

Setting ANSI colors using Python

print '\033[0;31mCRITICAL\033[0m'

Python function to set text colors using color name & bold flag

#!/usr/bin/env python
import sys
COLORS = (
    'BLACK', 'RED', 'GREEN', 'YELLOW',
    'BLUE', 'MAGENTA', 'CYAN', 'WHITE'
)

def color_text(text, color_name, bold=False):
    if color_name in COLORS:
        return '\033[{0};{1}m{2}\033[0m'.format(
            int(bold), COLORS.index(color_name) + 30, text)
    sys.stderr.write('ERROR: "{0}" is not a valid color.\n'.format(color_name))
    sys.stderr.write('VALID COLORS: {0}.\n'.format(', '.join(COLORS)))

# TESTS
if __name__ == '__main__':
    for bold in (False, True):
        for color_name in COLORS:
            print color_text('Example of {0}'.format(color_name), color_name, bold)
    print
    # test error handling
    color_text('TEST', 'SILVER')

$ less -Np "TABLE.*"

• Note the starting line number
• Search for “UNLOCK TABLES” (type ‘/’ in less to search)
• Note the line number found, this is the ending line number

$ mysql -u -p < sed -n <starting-line-num>,<ending-line-num>p

Example:

$ less -Np "TABLE.*user" dumpfile.sql
/UNLOCK TABLES
$ mysql -u jsmith -p logdata < sed -n 1034,1602p dumpfile.sql

How it works:

less -Np regex filename
    -N shows the line numbers
    -p opens the specified file at the specified regular expression

sed -n start,stopp
    -n supress automatic printing of pattern space
    start  - line number to start at
    stop - line number to stop at
    p - print

1. The mysqldump file will open up at the first line that starts with the word TABLE and contains your database table name. This shows the starting line number.
2. Since every mysqldump TABLE section ends with an UNLOCK TABLES line, search for it to determine the ending line number.
3. Use sed to extract the desired line numbers from the mysqldump file.
4. Redirect the output of sed to the input of MySQL to and to restore the database table, i.e. DROP, CREATE and populate.