pyARS - from python to ARS

Tutorial for usage of pyars.ARS

pyars - ARfunctions

This tutorial covers the C-style AR functions of the ARSystem API. In other words, it describes how to use the functions directly exposed from the API and what kind of C data structures you need to access them. There is another tutorial about the pythonic versions of the API functions available.

Working with entries

The most often used use case is probably working with entries. Let’s see, how we can retrieve, modify and create entries... We start with retrieving the entry with id ‘000000000000001’ from Form User. If this does not exist on your server, please modify this example accordingly to match your environment.

from pyars import ars, cars
from ctypes import *
ars = ars.ARS()
ars.Login('server', 'user', 'password')
# prepare the entry-id
tempArray = (cars.AREntryIdType * 1)()
tempArray[0][:] = '000000000000001\\0'\[:] # this needs to be 16 chars
entryId = cars.AREntryIdList(1, tempArray)
# error handling is left as an exercise to the reader :-)
entry = ars.ARGetEntry('User', entryId)

‘entry’ now contains an ARFieldValueList with three members, one for each field. Each ARFieldValueStruct contains a fieldId and a value struct; the value struct is a C union, that needs to be referenced depending on its type. In our example, we would like to print out the character fields:

for i in range(entry.numItems):
    if entry.fieldValueList[i].value.dataType == cars.AR_DATA_TYPE_CHAR:
    print '%d: %s' % (entry.fieldValueList[i].fieldId,
                      entry.fieldValueList[i].value.u.charVal)

Later in this tutorial you will lern a couple of helper functions that will ease working with the C structures...

Now let’s modify this users name...

# prepare the data structure for the field
tempArray = (cars.ARFieldValueStruct * 1)()
tempArray[0].fieldId = 8
tempArray[0].value = cars.ARValueStruct()
tempArray[0].value.dataType = cars.AR_DATA_TYPE_CHAR
tempArray[0].value.u.charVal = 'new user name\\0'
# ARSetEntry expects a list of such ARFieldValueStructs, even if you
# want to modify only a single field.
newFieldValue = cars.ARFieldValueList(1, tempArray)
res = ars.ARSetEntry('User', entryId, newFieldValue)
if res.errnr > 1:
    print 'ERROR modifying entry!'

A more detailed look at retrieving an entry

from pyars import ars

If you run into problems at this line, pyars cannot find the Remedy libraries to connect to your Remedy server. On windows, that means, that you don’t have a user tool or admin tool installed or strange entries in the registry. Make sure that the path to your libraries is contained in the system path.

ars = ars.ARS() # instantiate a session object
connection = ars.Login('server', 'user', 'password') # store the connection

If you run into problems with any API call at any point in time, you can have a look at the status message, that most often explains why an API call did not work. In order to give you easier access to the status messages, I created a function called statusText() that will return the strings of the message:

ars.statusText()

Now let’s continue with our example. There are different API calls to retrieve form entries from an ARsystem, let’s start with ARGetEntry, and let’s fetch the ‘User’ entry with entryid 1:

form = 'User'
# this is the entryid that we would like to retrieve:
entryid = 1
# import the constant definitions (hence the c)
from pyars import cars
# create an array to hold the entryid:
tempArray = (cars.AREntryIdType * 1)()
# store the entryid in the first element
tempArray[0][:] = ars.padEntryid(entryid)[:]
# create the appropriate data structure
idl = cars.AREntryIdList(1, tempArray)
# finally the call
entry = ars.ARGetEntry(form, idl)

Unfortunately, we need a lot of boiler plate code until we can simply call the API. But this is due to the fact that an AREntryId can be more complex (in case of joins, e.g.).

Now, that we have received something from the server, let’s have a look at that piece of data: When you enter

entry

you will see something along the following lines:

>>> <pyars._cars63.ARFieldValueList object at 0x00D07580>

How does this help us? First, whenever an ARsystem data structure is a list, it has the attribute numItems to tell us how many elements this list holds:

entry.numItems

In my case, it prints out 21L (that L is for the “long” data format). If you want to know how to access the single elements of the list, you need to look up the definition of the data structure in one of the _carsxx.py files that are specific for each ARSystem release, e.g. _cars63.py, _cars70.py etc. Looking up ARFieldValueList leads to the following lines (not necessarily next to each other...):

class ARFieldValueList(Structure):
    pass
ARFieldValueList._fields_ = [
    ('numItems', c_uint),
    ('fieldValueList', POINTER(ARFieldValueStruct)),
]

Don’t let the POINTER construct confuse you, this is just the way to say that this is the place to store an array of appropriate type and size.

Now you could dive into those data structures, but for the beginning let me help you with some helper functions so that you end with rather Pythonic data structures...

def convFieldValueList2Dict (obj):
    return dict([convFieldValueStruct2List(obj.fieldValueList[i])
                for i in range(obj.numItems)])
def convFieldValueStruct2List (obj):
    return [obj.fieldId, convValueStruct2Value(obj.value)]
cars.ARValueStruct._mapping_so_ = { cars.AR_DATA_TYPE_KEYWORD: 'u.keyNum',
cars.AR_DATA_TYPE_INTEGER: 'u.intVal',
cars.AR_DATA_TYPE_REAL: 'u.realVal',
cars.AR_DATA_TYPE_CHAR: 'u.charVal',
cars.AR_DATA_TYPE_DIARY: 'u.diaryVal',
cars.AR_DATA_TYPE_ENUM: 'u.enumVal',
cars.AR_DATA_TYPE_TIME: 'u.timeVal',
cars.AR_DATA_TYPE_BITMASK: 'u.maskVal',
cars.AR_DATA_TYPE_DECIMAL: 'u.decimalVal',
cars.AR_DATA_TYPE_ULONG: 'u.ulongVal',
cars.AR_DATA_TYPE_DATE: 'u.dateVal',
cars.AR_DATA_TYPE_TIME_OF_DAY: 'u.timeOfDayVal' }
def convValueStruct2Value(obj):
    try:
        if obj.dataType == cars.AR_DATA_TYPE_NULL:
            return None
        return eval('obj.%s' % (obj._mapping_so_ [obj.dataType]))
    except KeyError:
        print 'unknown ARValueStruct type!'
    return None

After all this, it’s now time to get to the values:

fieldValues = convFieldValueList2Dict(entry)

Now, you have a Python dictionary in fieldValues, where you can access all values based on fieldId:

print fieldValues[1]

And finally, we can free the memory of the data structures, that were returned from the ARsystem:

ars.ARFree(entry)

Retrieve a list of entries

If you don’t have the entry ids, you can search for entries (ARGetListEntry) or even get the data for the entries returned immediately (ARGetListEntryWithFields). Let’s concentrate on the second and retrieve the values for the fieldids 1, 5 and 8 for all entries in the form ‘User’:

# first we define a pseudo query that is always true; ignore None for now
query = ars.ARLoadARQualifierStruct('User', None, "1=1")
# the following is the list of fields we will retrieve for every result entry
fields = (1,5,8)
# a little helper function to fill the ARSystem data structure
def conv2EntryListFieldList(fieldList):
    tempArray = (cars.AREntryListFieldStruct * len(fieldList))()
    for i in range(len(fieldList)):
        if isinstance (fieldList[i], (long, int, str)):
            tempArray[i].fieldId = int(fieldList[i])
            tempArray[i].columnWidth = 10
            tempArray[i].separator = ''
    return cars.AREntryListFieldList (len(fieldList), tempArray)
# now use the helper function to convert the list of fields
arGetListFields = conv2EntryListFieldList(fields)
# now call the server
entries = ars.ARGetListEntryWithFields(form,
                        query,
                        arGetListFields)

entries now contains a tuple, consisting of an AREntryListFieldValueList and and integer showing the number of found entries that match the query on the form. On my server, entering “entries” and hitting return shows:

(<pyars._cars63.AREntryListFieldValueList object at 0x00B76AD0>, 3L)

Now it’s time to find the entry values buried in AREntryListFieldValueList: as you can see, this list is somehow related to the list we had before: ARFieldValueList. But this time, it is used with in an AREntryList, in other words, for every entry we will find a ARFieldValueList. We will need some helper functions again:

def convEntryListFieldValueList2Dict(obj):
    return dict([convEntryListFieldValueStruct2List(obj.entryList[i])
                for i in range(obj.numItems)])
def convEntryListFieldValueStruct2List(obj):
    return ['|'.join([obj.entryId.entryIdList[i].value for i in range(obj.entryId.numItems)]),
convFieldValueList2Dict(obj.entryValues.contents)]
# don't forget, that we only need to convert the first element of entries!
result = convEntryListFieldValueList2Dict(entries[0])

Now you have a Python dictionary that in the first hierarchy accesses the results by entryid, and in the second by fieldid, in other words:

>>> result['000000000000002']
{8L: 'Axel Seibert', 1L: '000000000000002', 5L: 'Demo'}
>>> result.keys()
['000000000000003', '000000000000002', '000000000000001']
>>> result['000000000000003'].keys()
[8L, 1L, 5L]

This was interactively, but of course, it’s very easy to go through all the elements:

for entry in result:
    print '''User: %s with entryId: %s''' % (result[entry][8],
                                            result[entry][1])

And when you’re done with the ARsytem data structures, as usual, it’s good behaviour to free the memory:

ars.ARFree(entries[0])

Please note: This example leaves the error handling as an exercise to the reader; typically, you should test ars.errnr after each API call for values greater than 1 (errors) and possibly 1 (warning). Only the error number 0 signals a successful call.

Retrieving an attachment into a buffer

Compared with retrieving an entry, this is quite easy:

tempArray = (cars.AREntryIdType * 1)()
tempArray [0][:] = ars.padEntryid('yourentryid') [:]
entryId = cars.AREntryIdList(1, tempArray)
# prepare a structure for the attachment, specify option BUFFER
loc = cars.ARLocStruct(cars.AR_LOC_BUFFER)
att = ars.ARGetEntryBLOB (form, entryId, 600010600, loc)
if att.u.buf.bufSize > 0:
    # assuming a plain text file
    print string_at(att.u.buf.buffer)

Retrieving an attachment as a file

Compared with retrieving an entry, this is quite easy:

tempArray = (cars.AREntryIdType * 1)()
tempArray [0][:] = ars.padEntryid('yourentryid') [:]
entryId = cars.AREntryIdList(1, tempArray)
# prepare a structure for the attachment, specify option BUFFER
loc = cars.ARLocStruct(cars.AR_LOC_FILENAME)
loc.u.filename = r'c:\temp\attachment.txt'
att = ars.ARGetEntryBLOB (form, entryId, 600010600, loc)

And the file is created in the filesystem automatically for you.

Get a list of schemas from an ARSystem

In this part we will retrieve the list of schemas from the ARSystem:

listSchema = ars.ARGetListSchema()   # fetch a list of all schema names
if ars.errnr < 2:
    for i in range(listSchema.numItems):
        print listSchema.nameList[i].value # .value is necessary to access the char pointer contents
        schema = ars.ARGetSchema(listSchema.nameList[i].value) # now fetch the detail information for this schema
        # schema is one of the convenience structures...
        # print out the help text for this schema:
        print schema.helpText
        ars.ARFree(schema) # free the memory
else:
    print "error retrieving list of schemas!"
    ars.ARFree(listSchema)

What did we do? Calling ARGetListSchema returns an ARNameList. If you lookup this structure in ar.h, you will find the following definition:

typedef struct ARNameList
{
    unsigned int   numItems;
    ARNameType    *nameList;
}  ARNameList;                /* list of 0 or more object names */

This has been translated to the following Python structure:

class ARNameList(Structure):
    _fields_ = [('numItems', c_uint),
                ('nameList', POINTER(ARNameType))]

So, if you have an object of type ARNameList with name schemas, you can access those member variables as you would in C:

schemas.numItems, schemas.nameList[index]

There is only one catch: When you want to access the contents of a char pointer (as opposed to the pointer object itself), you need to add a ”.value” at the end:

schemas.nameList[i].value

So, if there was no error retrieving the list of schema names, we can iterate over the resulting name list and try to fetch detailed schema information for each schema.

So far, this looks very much like C code translated to python. How can we profit from python’s features? Look at the following example to see how easy it is to convert the C data structure into native Python structures and then to use Python’s libraries to work on them.

schemaList = [schemas.nameList[i].value for i in range(schemas.numItems)]
schemaList.sort()
for schema in schemaList:
    print schema

As there is no nice structure defined by Remedy/BMC that would hold all relevant schema information (as retrieved by ARGetSchema) we came up with our own structure:

class ARSchema(Structure):
    _fields_ = [("name", cars.ARNameType),
("schema", cars.ARCompoundSchema),
("schemaInheritanceList", cars.ARSchemaInheritanceList),
("groupList", cars.ARPermissionList),
("admingrpList", cars.ARInternalIdList),
("getListFields",  cars.AREntryListFieldList),
("sortList", cars.ARSortList),
("indexList", cars.ARIndexList),
("archiveInfo", cars.ARArchiveInfoStruct),
("defaultVui", cars.ARNameType),
("helpText", c_char_p),
("timestamp", cars.ARTimestamp),
("owner", cars.ARAccessNameType),
("lastChanged", cars.ARAccessNameType),
("changeDiary", c_char_p),
("objPropList", cars.ARPropList)]

So, in a sense, those additional structures allow us to be a litte bit more pythonic than with the usual data structures. However, with this level of abstraction, pyars is still far away from being a pythonic way to access ARSystems. I could think of many ways to improve things (e.g. automatic conversion of python data structures to ARSystem C data structures, or exceptions instead of error numbers). Maybe with the help of the community, it would be possible to implement such a pythonic convenience layer on top of pyars.ars and pyars.cmdb?

To demonstrate how you can use the lookup dictionary (assuming, you set up a connection with your ARS):

from pyars import cars
field = ars.ARGetField('User', 7)
print 'field %s(%d) of type %s' % (field.fieldName,
                                   field.fieldId,
                           cars.ars_const['AR_DATA_TYPE'][field.dataType])
ars.ARFree(field)

In the file cars.py you will find the definition of this dictionary. So if you want to print out something, go in there and search for the constant.

At the end of your session, it’s always a good idea to logoff:

ars.Logoff()

Python makes it easy to log off from the system in any case:

try:
    ... # some code
finally:
    ars.Logoff()

Check permissions for a schema

Let’s look at another example of the ARSystem data structures. Let’s fetch the schema information and go through the permissions...

schema = ars.ARGetSchema('User')

Now we have an ARSchema struct in variable schema; looking at the definition above, you can see that the fourth attribute (groupList) contains a permissionlist. Let’s have a look at this list:

ARPermissionList._fields_ = [
# ar.h 2605
('numItems', c_uint),
('permissionList', POINTER(ARPermissionStruct)),
]
ARPermissionStruct._fields_ = [
# ar.h 2598
('groupId', ARInternalId),
('permissions', c_uint),
]

The list is constructed as usual, and the actual structure only contains two members, so iterating and printing the values should be easy:

for i in range(schema.groupList.numItems):
    print 'line #%d: groupId: %d has permissions: %d' % (
            i,
            arpermissionList.permissionList[i].groupId,
            arpermissionList.permissionList[i].permissions)

This results in a basic output of the groupIds and associated permissions. However, if we want to lookup the groupId, we need to access the following data structures (as returned by ARGetListGroup).

ARGroupInfoList._fields_ = [
# ar.h 2638
('numItems', c_uint),
('groupList', POINTER(ARGroupInfoStruct)),
]
ARGroupInfoStruct._fields_ = [
# ar.h 2628
('groupId', ARInternalId),
('groupType', c_uint),
('groupName', ARAccessNameList),
('groupCategory', c_uint),
]

So let’s fetch all group information from the server, build a dictionary to lookup the group id’s name (it’s a bit complicated, as a single groupid can map to more than one name, as groupName points to an ARAccessNameList!). The permissions we can lookup in the lookup dictionary cars.ars_const[‘AR_PERMISSIONS’].

groupList = ars.ARGetListGroup()
lookupGroups = {} # initialize the dictionary
for i in range(groupList.numItems):
    groupNames = [groupList.groupList[i].groupName.nameList[j].value
    for j in range(groupList.groupList[i].groupName.numItems)]
        lookupGroups [groupList.groupList[i].groupId] = groupNames

Now we have everything to have a nicer printout:

for i in range(schema.groupList.numItems):
    print 'line #%d: group: %s (%d) has permissions: %s' % (
            i,
            lookupGroups[schema.groupList.permissionList[i].groupId],
            schema.groupList.permissionList[i].groupId,
            cars.ars_const['AR_PERMISSIONS'][schema.groupList.permissionList[i].permissions])

And finally it’s time to logoff....

ars.Logoff()

Working with the XML functions

So far, not all XML functions have been implemented. However, with those functions I made sure that you can feed them the output of the corresponding ARGet functions (e.g. that you can feed the result of ARGetSchema to ARSetSchemaToXML).

Writing out the definition of a schema to XML:

from pyars import ars, cars
a = ars.ARS()
a.Login('server:port', 'user', 'password')
schema = a.ARGetSchema('User')
# now fetch all information about the fields
# you need to hand over all lists that you want to be filled...
# however, the wrapper for ARGetMultipleFields will construct
# an ARFieldInfoList and return that...
fieldName = cars.ARNameList()
fieldMap = cars.ARFieldMappingList()
dataType = cars.ARUnsignedIntList()
fields = a.ARGetMultipleFields('User', fieldName = fieldName,
fieldMap = fieldMap,
dataType = dataType)
# fetch information about the views
vuis = a.ARGetMultipleVUIs('User')
# and write out everything to an XML string:
result = a.ARSetSchemaToXML('User', 0, schema.schema, schema.groupList,
schema.admingrpList, schema.getListFields,
schema.sortList, schema.indexList, schema.archiveInfo,
schema.defaultVui, 0, 0, 0, fields, vuis, schema.owner,
schema.lastChanged, schema.timestamp, schema.helpText,
schema.changeDiary, schema.objPropList, 0)
file=open(r'd:\user.xml', 'w')
file.write(result)
file.close()

The example above was written for a 6.3 server. In version 7.0, the schema definition was extended by the auditInfo. Now, if you run the above example against a 7.0 server with 7.0 dlls installed on your machine, you will run into problems, because one argument is missing, and what’s worse, following arguments are off by one position.

To remedy this situation, you can use positional arguments:

result = ar.ARSetSchemaToXML(schemaName = 'User',
xmlDocHdrFtrFlag = False,
compoundSchema=schema.schema,
permissionList=schema.groupList,
subAdminGrpList=schema.admingrpList,
getListFields=schema.getListFields,
sortList=schema.sortList,
indexList=schema.indexList,
archiveInfo = schema.archiveInfo,
defaultVui = schema.defaultVui,
nextFieldID = 0,
coreVersion = 0,
upgradeVersion=0,
fieldInfoList=fields,
vuiInfoList=vuis,
owner=schema.owner,
lastModifiedBy=schema.lastChanged,
modifiedDate=schema.timestamp,
helpText=schema.helpText,
changeHistory=schema.changeDiary,
objPropList=schema.objPropList,
arDocVersion = 0)

With this code, python recognizes that the argument auditInfo is not handed over and will use the default value, as defined in pyars.ars (None).

I have to admit that this is a lot of typing; to minimize the risk of wrong argument types, I’ve started to augment many functions (especially the XML functions) with asserts, that check the type of the argument.

Please note the different calling convention, when compared with the original C API functions: whereas the C API functions expect an ARXMLDoc structure as the first argument, the wrapper functions take care of this structure, themselves. Instead, the wrapper expects the name of the object as the first parameter (not third), and will return an xml string (or None in case of failure).

How to proceed

Generally speaking, you need to follow those steps:

• Identify which API function you want to call
• Identify which data structures this API call expects
• Prepare those data structures
• Call the API function
• Identify which data structures this API call returns; have a look at the examples in ars_test.py and cmdb_test.py, respectively. You will find many helpful code snippets in there.
• Have a look at the unit tests (ars_test.py and cmdb_test.py) which provide many useful examples.
• Give us feedback, possibly submit code, patches.