pyARS - from python to ARS
##########################

===============================
Tutorial for usage of pyars.ARS
===============================

pyars - ARfunctions
###################

This tutorial covers the C-style AR functions of the ARSystem API. In
other words, it describes how to use the functions directly exposed from
the API and what kind of C data structures you need to access them. There is
another `tutorial about the pythonic versions of the
API functions <tutorial.html>`_ available.

Working with entries
####################

The most often used use case is probably working with entries. Let's see,
how  we can retrieve, modify and create entries... We start with retrieving
the entry with id '000000000000001' from Form User. If this does not
exist on your server, please modify this example accordingly to match your
environment.

::

    from pyars import ars, cars
    from ctypes import *
    ars = ars.ARS()
    ars.Login('server', 'user', 'password')
    # prepare the entry-id
    tempArray = (cars.AREntryIdType * 1)()
    tempArray[0][:] = '000000000000001\\0'\[:] # this needs to be 16 chars
    entryId = cars.AREntryIdList(1, tempArray)
    # error handling is left as an exercise to the reader :-)
    entry = ars.ARGetEntry('User', entryId)

'entry' now contains an ARFieldValueList with three members, one for each field.
Each ARFieldValueStruct contains a fieldId and a value struct; the value struct
is a C union, that needs to be referenced depending on its type. In our example,
we would like to print out the character fields:
::

    for i in range(entry.numItems):
        if entry.fieldValueList[i].value.dataType == cars.AR_DATA_TYPE_CHAR:
        print '%d: %s' % (entry.fieldValueList[i].fieldId,
                          entry.fieldValueList[i].value.u.charVal)

Later in this tutorial you will lern a couple of helper functions
that will ease working with the C structures...

Now let's modify this users name...
::

    # prepare the data structure for the field
    tempArray = (cars.ARFieldValueStruct * 1)()
    tempArray[0].fieldId = 8
    tempArray[0].value = cars.ARValueStruct()
    tempArray[0].value.dataType = cars.AR_DATA_TYPE_CHAR
    tempArray[0].value.u.charVal = 'new user name\\0'
    # ARSetEntry expects a list of such ARFieldValueStructs, even if you
    # want to modify only a single field.
    newFieldValue = cars.ARFieldValueList(1, tempArray)
    res = ars.ARSetEntry('User', entryId, newFieldValue)
    if res.errnr > 1:
        print 'ERROR modifying entry!'

A more detailed look at retrieving an entry
###########################################

::

    from pyars import ars

If you run into problems at this line, pyars cannot find the Remedy libraries to
connect to your Remedy server. On windows, that means, that you don't have a
user tool or admin tool installed or strange entries in the registry.
Make sure that the path to your libraries is contained in the system path.
::

    ars = ars.ARS() # instantiate a session object
    connection = ars.Login('server', 'user', 'password') # store the connection

If you run into problems with any API call at any point in time, you can have
a look at the status message, that most often explains why an API call did
not work. In order to give you easier access to the status messages, I created a
function called statusText() that will return the strings of the message:
::

    ars.statusText()

Now let's continue with our example. There are different API calls to retrieve
form entries from an ARsystem, let's start with ARGetEntry, and let's fetch
the 'User' entry with entryid 1:
::

    form = 'User'
    # this is the entryid that we would like to retrieve:
    entryid = 1
    # import the constant definitions (hence the c)
    from pyars import cars
    # create an array to hold the entryid:
    tempArray = (cars.AREntryIdType * 1)()
    # store the entryid in the first element
    tempArray[0][:] = ars.padEntryid(entryid)[:]
    # create the appropriate data structure
    idl = cars.AREntryIdList(1, tempArray)
    # finally the call
    entry = ars.ARGetEntry(form, idl)

Unfortunately, we need a lot of boiler plate code until we can simply call the
API. But this is due to the fact that an AREntryId can be more complex (in
case of joins, e.g.).

Now, that we have received something from the server, let's have a look
at that piece of data: When you enter
::

    entry

you will see something along the following lines:
::

    >>> <pyars._cars63.ARFieldValueList object at 0x00D07580>

How does this help us? First, whenever an ARsystem data structure is a list,
it has the attribute numItems to tell us how many elements this list holds:
::

    entry.numItems

In my case, it prints out 21L (that L is for the "long" data format). If you
want to know how to access the single elements of the list, you need
to look up the definition of the data structure in one of the _carsxx.py files
that are specific for each ARSystem release, e.g. _cars63.py, _cars70.py etc.
Looking up ARFieldValueList leads to the following lines (not necessarily
next to each other...):
::

    class ARFieldValueList(Structure):
        pass
    ARFieldValueList._fields_ = [
        ('numItems', c_uint),
        ('fieldValueList', POINTER(ARFieldValueStruct)),
    ]

Don't let the POINTER construct confuse you, this is just the way to say
that this is the place to store an array of appropriate type and size.

Now you could dive into those data structures, but for the beginning let
me help you with some helper functions so that you end with rather
Pythonic data structures...
::

    def convFieldValueList2Dict (obj):
        return dict([convFieldValueStruct2List(obj.fieldValueList[i])
                    for i in range(obj.numItems)])
    def convFieldValueStruct2List (obj):
        return [obj.fieldId, convValueStruct2Value(obj.value)]
    cars.ARValueStruct._mapping_so_ = { cars.AR_DATA_TYPE_KEYWORD: 'u.keyNum',
    cars.AR_DATA_TYPE_INTEGER: 'u.intVal',
    cars.AR_DATA_TYPE_REAL: 'u.realVal',
    cars.AR_DATA_TYPE_CHAR: 'u.charVal',
    cars.AR_DATA_TYPE_DIARY: 'u.diaryVal',
    cars.AR_DATA_TYPE_ENUM: 'u.enumVal',
    cars.AR_DATA_TYPE_TIME: 'u.timeVal',
    cars.AR_DATA_TYPE_BITMASK: 'u.maskVal',
    cars.AR_DATA_TYPE_DECIMAL: 'u.decimalVal',
    cars.AR_DATA_TYPE_ULONG: 'u.ulongVal',
    cars.AR_DATA_TYPE_DATE: 'u.dateVal',
    cars.AR_DATA_TYPE_TIME_OF_DAY: 'u.timeOfDayVal' }
    def convValueStruct2Value(obj):
        try:
            if obj.dataType == cars.AR_DATA_TYPE_NULL:
                return None
            return eval('obj.%s' % (obj._mapping_so_ [obj.dataType]))
        except KeyError:
            print 'unknown ARValueStruct type!'
        return None

After all this, it's now time to get to the values:
::

    fieldValues = convFieldValueList2Dict(entry)

Now, you have a Python dictionary in fieldValues, where you can access all
values based on fieldId:
::

    print fieldValues[1]

And finally, we can free the memory of the data structures, that were
returned from the ARsystem:
::

    ars.ARFree(entry)

Retrieve a list of entries
##########################

If you don't have the entry ids, you can search for entries (ARGetListEntry) or
even get the data for the entries returned immediately (ARGetListEntryWithFields).
Let's concentrate on the second and retrieve the values for the fieldids 1, 5
and 8 for all entries in the form 'User':
::

    # first we define a pseudo query that is always true; ignore None for now
    query = ars.ARLoadARQualifierStruct('User', None, "1=1")
    # the following is the list of fields we will retrieve for every result entry
    fields = (1,5,8)
    # a little helper function to fill the ARSystem data structure
    def conv2EntryListFieldList(fieldList):
        tempArray = (cars.AREntryListFieldStruct * len(fieldList))()
        for i in range(len(fieldList)):
            if isinstance (fieldList[i], (long, int, str)):
                tempArray[i].fieldId = int(fieldList[i])
                tempArray[i].columnWidth = 10
                tempArray[i].separator = ''
        return cars.AREntryListFieldList (len(fieldList), tempArray)
    # now use the helper function to convert the list of fields
    arGetListFields = conv2EntryListFieldList(fields)
    # now call the server
    entries = ars.ARGetListEntryWithFields(form,
                            query,
                            arGetListFields)

entries now contains a tuple, consisting of an AREntryListFieldValueList and
and integer showing the number of found entries that match the query on the form.
On my server, entering "entries" and hitting return shows:
::

    (<pyars._cars63.AREntryListFieldValueList object at 0x00B76AD0>, 3L)

Now it's time to find the entry values buried in AREntryListFieldValueList: as
you can see, this list is somehow related to the list we had before: ARFieldValueList.
But this time, it is used with in an AREntryList, in other words, for every
entry we will find a ARFieldValueList. We will need some helper functions again:
::

    def convEntryListFieldValueList2Dict(obj):
        return dict([convEntryListFieldValueStruct2List(obj.entryList[i])
                    for i in range(obj.numItems)])
    def convEntryListFieldValueStruct2List(obj):
        return ['|'.join([obj.entryId.entryIdList[i].value for i in range(obj.entryId.numItems)]),
    convFieldValueList2Dict(obj.entryValues.contents)]
    # don't forget, that we only need to convert the first element of entries!
    result = convEntryListFieldValueList2Dict(entries[0])

Now you have a Python dictionary that in the first hierarchy accesses the
results by entryid, and in the second by fieldid, in other words:
::

    >>> result['000000000000002']
    {8L: 'Axel Seibert', 1L: '000000000000002', 5L: 'Demo'}
    >>> result.keys()
    ['000000000000003', '000000000000002', '000000000000001']
    >>> result['000000000000003'].keys()
    [8L, 1L, 5L]

This was interactively, but of course, it's very easy to go through all
the elements:
::

    for entry in result:
        print '''User: %s with entryId: %s''' % (result[entry][8],
                                                result[entry][1])

And when you're done with the ARsytem data structures, as usual, it's good
behaviour to free the memory:
::

    ars.ARFree(entries[0])

Please note: This example leaves the error handling as an exercise to the reader;
typically,
you should test ars.errnr after each API call for values greater than 1 (errors)
and possibly 1 (warning). Only the error number 0 signals a successful call.

Retrieving an attachment into a buffer
######################################

Compared with retrieving an entry, this is quite easy:
::

    tempArray = (cars.AREntryIdType * 1)()
    tempArray [0][:] = ars.padEntryid('yourentryid') [:]
    entryId = cars.AREntryIdList(1, tempArray)
    # prepare a structure for the attachment, specify option BUFFER
    loc = cars.ARLocStruct(cars.AR_LOC_BUFFER)
    att = ars.ARGetEntryBLOB (form, entryId, 600010600, loc)
    if att.u.buf.bufSize > 0:
        # assuming a plain text file
        print string_at(att.u.buf.buffer)

Retrieving an attachment as a file
##################################

Compared with retrieving an entry, this is quite easy:
::

    tempArray = (cars.AREntryIdType * 1)()
    tempArray [0][:] = ars.padEntryid('yourentryid') [:]
    entryId = cars.AREntryIdList(1, tempArray)
    # prepare a structure for the attachment, specify option BUFFER
    loc = cars.ARLocStruct(cars.AR_LOC_FILENAME)
    loc.u.filename = r'c:\temp\attachment.txt'
    att = ars.ARGetEntryBLOB (form, entryId, 600010600, loc)

And the file is created in the filesystem automatically for you.

Get a list of schemas from an ARSystem
######################################

In this part we will retrieve the list of schemas from the ARSystem:
::

    listSchema = ars.ARGetListSchema()   # fetch a list of all schema names
    if ars.errnr < 2:
        for i in range(listSchema.numItems):
            print listSchema.nameList[i].value # .value is necessary to access the char pointer contents
            schema = ars.ARGetSchema(listSchema.nameList[i].value) # now fetch the detail information for this schema
            # schema is one of the convenience structures...
            # print out the help text for this schema:
            print schema.helpText
            ars.ARFree(schema) # free the memory
    else:
        print "error retrieving list of schemas!"
        ars.ARFree(listSchema)

What did we do? Calling ARGetListSchema returns an ARNameList. If you
lookup this structure in ar.h, you
will find the following definition:
::

    typedef struct ARNameList
    {
        unsigned int   numItems;
        ARNameType    *nameList;
    }  ARNameList;                /* list of 0 or more object names */

This has been translated to the following Python structure:
::

    class ARNameList(Structure):
        _fields_ = [('numItems', c_uint),
                    ('nameList', POINTER(ARNameType))]

So, if you have an object of type ARNameList with name schemas, you can access
those member variables
as you would in C:
::

    schemas.numItems, schemas.nameList[index]

There is only one catch: When you want to access the contents of a char
pointer (as opposed
to the pointer object itself), you need to add a ".value" at the end:
::

    schemas.nameList[i].value

So, if there was no error retrieving the list of schema names, we can iterate
over the resulting name list and try to fetch detailed schema information
for each schema.

So far, this looks very much like C code translated to python. How can we
profit from python's features? Look at the following example to see how easy it
is to convert the C data structure into native Python structures and then to
use Python's libraries to work on them.
::

    schemaList = [schemas.nameList[i].value for i in range(schemas.numItems)]
    schemaList.sort()
    for schema in schemaList:
        print schema

As there is no nice structure defined by Remedy/BMC that
would hold all relevant schema information (as retrieved by ARGetSchema) we
came up with our own structure:
::

    class ARSchema(Structure):
        _fields_ = [("name", cars.ARNameType),
    ("schema", cars.ARCompoundSchema),
    ("schemaInheritanceList", cars.ARSchemaInheritanceList),
    ("groupList", cars.ARPermissionList),
    ("admingrpList", cars.ARInternalIdList),
    ("getListFields",  cars.AREntryListFieldList),
    ("sortList", cars.ARSortList),
    ("indexList", cars.ARIndexList),
    ("archiveInfo", cars.ARArchiveInfoStruct),
    ("defaultVui", cars.ARNameType),
    ("helpText", c_char_p),
    ("timestamp", cars.ARTimestamp),
    ("owner", cars.ARAccessNameType),
    ("lastChanged", cars.ARAccessNameType),
    ("changeDiary", c_char_p),
    ("objPropList", cars.ARPropList)]

So, in a sense, those additional structures allow us to be a litte bit more
pythonic than with the usual data structures. However, with this level
of abstraction, pyars is still far away from being a pythonic way to access
ARSystems. I could think of many ways to improve things (e.g. automatic
conversion of python data structures to ARSystem C data structures, or
exceptions instead of error numbers). Maybe with the help of the community, it
would be possible to implement such a pythonic convenience layer on top of
pyars.ars and pyars.cmdb?

To demonstrate how you can use the lookup dictionary
(assuming, you set up
a connection with your ARS):
::

    from pyars import cars
    field = ars.ARGetField('User', 7)
    print 'field %s(%d) of type %s' % (field.fieldName,
                                       field.fieldId,
                               cars.ars_const['AR_DATA_TYPE'][field.dataType])
    ars.ARFree(field)

In the file cars.py you will find the definition of this dictionary. So if
you want to print out something, go in there and search for the constant.

At the end of your session, it's always a good idea to logoff:
::

    ars.Logoff()

Python makes it easy to log off from the system in any case:
::

    try:
        ... # some code
    finally:
        ars.Logoff()

Check permissions for a schema
##############################

Let's look at another example of the ARSystem data structures. Let's
fetch the schema information and go through the permissions...
::

    schema = ars.ARGetSchema('User')

Now we have an ARSchema struct in variable schema; looking at the definition
above, you can see that the fourth attribute (groupList) contains a permissionlist.
Let's have a look at this list:
::

    ARPermissionList._fields_ = [
    # ar.h 2605
    ('numItems', c_uint),
    ('permissionList', POINTER(ARPermissionStruct)),
    ]
    ARPermissionStruct._fields_ = [
    # ar.h 2598
    ('groupId', ARInternalId),
    ('permissions', c_uint),
    ]

The list is constructed as usual, and the actual structure only contains two
members, so iterating and printing the values should be easy:
::

    for i in range(schema.groupList.numItems):
        print 'line #%d: groupId: %d has permissions: %d' % (
                i,
                arpermissionList.permissionList[i].groupId,
                arpermissionList.permissionList[i].permissions)

This results in a basic output of the groupIds and associated permissions.
However, if we want to lookup the groupId, we need to access
the following data structures (as returned by ARGetListGroup).
::

    ARGroupInfoList._fields_ = [
    # ar.h 2638
    ('numItems', c_uint),
    ('groupList', POINTER(ARGroupInfoStruct)),
    ]
    ARGroupInfoStruct._fields_ = [
    # ar.h 2628
    ('groupId', ARInternalId),
    ('groupType', c_uint),
    ('groupName', ARAccessNameList),
    ('groupCategory', c_uint),
    ]

So let's fetch all group information from the server, build a dictionary to
lookup the group id's name (it's a bit complicated, as a single groupid can
map to more than one name, as groupName points to an ARAccessNameList!).
The permissions we can lookup in the lookup dictionary
cars.ars_const['AR_PERMISSIONS'].
::

    groupList = ars.ARGetListGroup()
    lookupGroups = {} # initialize the dictionary
    for i in range(groupList.numItems):
        groupNames = [groupList.groupList[i].groupName.nameList[j].value
        for j in range(groupList.groupList[i].groupName.numItems)]
            lookupGroups [groupList.groupList[i].groupId] = groupNames

Now we have everything to have a nicer printout:
::

    for i in range(schema.groupList.numItems):
        print 'line #%d: group: %s (%d) has permissions: %s' % (
                i,
                lookupGroups[schema.groupList.permissionList[i].groupId],
                schema.groupList.permissionList[i].groupId,
                cars.ars_const['AR_PERMISSIONS'][schema.groupList.permissionList[i].permissions])

And finally it's time to logoff....
::

    ars.Logoff()

Working with the XML functions
##############################

So far, not all XML functions have been implemented. However, with those
functions I made sure that you can feed them the output of
the corresponding ARGet functions (e.g. that you can feed the result
of ARGetSchema to ARSetSchemaToXML).

Writing out the definition of a schema to XML:
::

    from pyars import ars, cars
    a = ars.ARS()
    a.Login('server:port', 'user', 'password')
    schema = a.ARGetSchema('User')
    # now fetch all information about the fields
    # you need to hand over all lists that you want to be filled...
    # however, the wrapper for ARGetMultipleFields will construct
    # an ARFieldInfoList and return that...
    fieldName = cars.ARNameList()
    fieldMap = cars.ARFieldMappingList()
    dataType = cars.ARUnsignedIntList()
    fields = a.ARGetMultipleFields('User', fieldName = fieldName,
    fieldMap = fieldMap,
    dataType = dataType)
    # fetch information about the views
    vuis = a.ARGetMultipleVUIs('User')
    # and write out everything to an XML string:
    result = a.ARSetSchemaToXML('User', 0, schema.schema, schema.groupList,
    schema.admingrpList, schema.getListFields,
    schema.sortList, schema.indexList, schema.archiveInfo,
    schema.defaultVui, 0, 0, 0, fields, vuis, schema.owner,
    schema.lastChanged, schema.timestamp, schema.helpText,
    schema.changeDiary, schema.objPropList, 0)
    file=open(r'd:\user.xml', 'w')
    file.write(result)
    file.close()

The example above was written for a 6.3 server. In version 7.0, the schema
definition was extended by the auditInfo. Now, if you run the above example
against a 7.0 server with 7.0 dlls installed on your machine, you will run
into problems, because one argument is missing, and what's worse, following
arguments are off by one position.

To remedy this situation, you can use positional arguments:
::

    result = ar.ARSetSchemaToXML(schemaName = 'User',
    xmlDocHdrFtrFlag = False,
    compoundSchema=schema.schema,
    permissionList=schema.groupList,
    subAdminGrpList=schema.admingrpList,
    getListFields=schema.getListFields,
    sortList=schema.sortList,
    indexList=schema.indexList,
    archiveInfo = schema.archiveInfo,
    defaultVui = schema.defaultVui,
    nextFieldID = 0,
    coreVersion = 0,
    upgradeVersion=0,
    fieldInfoList=fields,
    vuiInfoList=vuis,
    owner=schema.owner,
    lastModifiedBy=schema.lastChanged,
    modifiedDate=schema.timestamp,
    helpText=schema.helpText,
    changeHistory=schema.changeDiary,
    objPropList=schema.objPropList,
    arDocVersion = 0)

With this code, python recognizes that the argument auditInfo is
not handed over and will use the default value, as defined in pyars.ars
(None).

I have to admit that this is a lot of typing; to minimize the risk of
wrong argument types, I've started to augment many functions (especially
the XML functions) with asserts, that check the type of the argument.

Please note the different calling convention, when compared with
the original C API functions: whereas the C API functions expect an ARXMLDoc
structure as the first argument, the wrapper functions take care of this
structure, themselves. Instead, the wrapper expects the name of the object
as the first parameter (not third), and will return an xml string (or None
in case of failure).

How to proceed
##############

Generally speaking, you need to follow those steps:

- Identify which API function you want to call

- Identify which data structures this API call expects

- Prepare those data structures

- Call the API function

- Identify which data structures this API call returns; have a look at the
  examples in ars_test.py and cmdb_test.py, respectively. You will
  find many helpful code snippets in there.

- Have a look at the unit tests (ars_test.py and cmdb_test.py) which
  provide many useful examples.

- Give us feedback, possibly submit code, patches.