Goal Attainment
Home Day Support Services Career Opportunities Shared Living Case Management Residential Services Youth Services Newsletter Training Calendar Goal Attainment Executive Director Billing Compliance Community Center Testimonials Employment

 

Assessing Goal Attainment at Mobius


Beginning in 2006, Mobius began phasing-in the assessment of attainment of goals identified within person-centered plans. Currently, over 40 individuals have had goals assessed utilizing a process which was modified from scales used with other service populations. The organization was seeking a method for assessing the effectiveness of individual plans and services in supporting positive outcomes for individuals.


Goal attainment scaling provides a means to assess the amount of relative change by considering information from any combination of measurement, observation and/or reporting sources. The process does not determine whether goals are relevant (meaningful to a person's wishes and
needs) or reasonably challenging for an individual. Those assurances can only be derived from qualitative review of an organization's person-centered planning process, and/or specific review of an individual plan. A manuscript, 'Assessing Goal Attainment for Quality Improvement (Lawlor & York),' describing the approach  has been published in the September, 2007 issue of the Journal of Intellectual Disabilities.


Assessment of goal attainment is most readily useful for designing individual plans on a case-by-case basis. The application of data resulting from aggregated ratings is more challenging to demonstrate at the program and organizational levels.   However, it can be useful for monitoring or troubleshooting purposes when applied with other organizational information.


Mobius currently utilizes a variety of information sources, in addition to assessing goal attainment, to assist with quality improvement efforts at the individual and program levels. The following instruments are utilized.


The National Core Indicator Project: A national instrument administered by provider agencies in Maine according to a specific schedule determined by the State. Includes key outcome indicators for service recipients within participating state systems.


Consumer Satisfaction Survey: Implemented annually by Mobius to determine consumer satisfaction with services and processes.


Person-Centered Planning Tools: Includes extensive strength and needs assessment, risk assessment, and any of fifteen optional tools.


Goal Attainment Data Analysis

Initial analysis of goal attainment data has been limited to descriptive statistics and preliminary calculations for coefficients of rater reliability. Next steps for data analysis include: 1) expanded collection of goal attainment data and 2) identification of variables for correlation with goal attainment data.








The first bar graph above is a simple presentation of distribution of goals assessed by domain type.

The second graph depicts mean ratings by rater and domain types.

The graph directly above is an error bar graph that shows the average domain ratings for three rater types: consumers (RCLIENT); program managers (RMAN) and independent case managers (RISC). Family members responses were too few in number to be included in this presentation. Some comments about this graphic follow.

        - The mean is in the middle of each bar displayed.

        - The bars represent a 95 confidence level.

        - The length of the bars is a function of numbers of observations and the variability of the observations (ratings). As the number of observations increase and the variability of the observations decrease, the bars get smaller.

        - When the bars overlap, that means that there are not statistically significant differences between the means of the raters. When the bars do not overlap, that means that there is a significance mean difference (DeSisto, 2007).

The graph shows visually that the primary difference among the three rater types is between the RISC and the RCLIENT goal attainment ratings in the social domain. This consistently reflects other findings in the literature which suggest that goals and outcomes within the social/inclusion/community-type domains are more difficult to predict, given the limited  control that support programs have on community  environments. (Gardiner & Carran, 2005; Krispjin, et. AL, 1999; Leon et. al., 1999).

Updated: July, 2011

The Mobius goal analysis program produced it’s most significant work to date with the completion of a research project jointly supported by the Maine Health Access Foundation and the Maine Office of Adults with Cognitive and Physical Disabilities. The following final report was submitted in April, 2011.

Program Goals and Objectives: The goal of the project was to collect and study data contained within service plans from two agencies operating within DHHS Region II (central), and from samples in DHHS offices in Region 1 (south) and Region III (north). Specific objectives of the project were to 1) implement assessment of goal attainment in the Region II agencies, 2) collect goal statement samples from all sources; 3) classify goal statements according to health activity and other goal and service recipient characteristics, and 4) analyze resulting data for frequencies distribution and correlations, One purpose was to identify the degree of goal activity that addresses individual health issues, and to explore associations between and among goal and subject characteristics based on available data.

Key Program Activities

February 1, 2010: Internal data collection initiated.
February 8, 2010: First meeting of advisory group.
March 1, 2010: Agreements completed for external data collection.
                            Data classification protocols identified and classification
                            begun.
December 15, 2010: Initial statistical analyses completed.
March 10, 2011: Final meeting of advisory group. Additional analyses recommended and  
                           completed.
April 1, 2011:   Presentation to New England TASH conference, Worcester, MA.
April 26, 2011: Presentation to senior management of Office of  Adults with Cognitive
                           and Physical Disabilities (OACPD), DHHS, Augusta, ME. Priorities for  
                           further data analyses received.
June 9, 2011:   Presentation to occur of communication data at non-traditional  
                           communication conference, Wells, ME

Collaborators: Additional project staff included: Emma Stephenson, Mobius Quality Manager, for data classification; Jacob Lowell, MA Statistics, for data analyses; and  Romy Spitz, Ph.D. and  Michael DeSisto, Ph.D. for consultation on project design. Other collaborators were critical to the program and participated in several ways. Initial input was received from the project advisory group to advise priorities and from a second meeting convened to review early study results and recommend areas for further study. Advisory group participants included: Jane Gallivan, the Director of the Office of Adults with Cognitive and Physical Disabilities (OACPD); Jay Yoe, Director of DHHS Quality Improvement; Margaret Rode, OACPD Director of Quality Management; Laurie Kimball, OACPD Regional Quality Improvement Staff; and project staff.  Implementation included extensive collaboration between Mobius project staff and The Progress Center, Norway, Maine and coordination with managers of OACPD regional operations in the Caribou and Portland offices of DHHS.

The implementation timeline ran from 2/1/2010 to 4/31/2011 (as extended). All activities described above (advisory input, design detailing, data collection, analysis) occurred within time frames that enabled the project to complete key activities on schedule.

Data was collected from recipient plans and BMS 99 forms. Under advisement from the DHHS Office of Quality Improvement, a project-specific coding method was used to protect the identity of service recipients.
 
Project Outcomes:

1700 goal statements from 133 service plans obtained from the 3 State service regions were classified according to 18 variables representing 6 goal and 12 individual characteristics.

   Chi Square Analyses

Chi square analyses were conducted using the following dependent variables and classifications. 

   Goal Characteristics

  1. Goal Domain Type:  Physical health or  Other
  2. Goal Skill  Content: Indicated  or Not Indicated
  3. Goal Community Participation: Indicated or Not indicated
  4. Goal Communication Content:  Present & Participating, Present or Not Present
  5. Goal Attainment: Above the Mean Rating or Below the Mean Rating
  6. Physical Health Sub-Domains: Fitness, Hygiene, Medical Care, Nutrition, Personal Safety, Other

    Individual Characteristics

  1.  Health Maintenance Ability: Independent, Training Needed or Support Needed
  2.  Guardianship Status: Self or  Other
  3.  Medication Administration Ability: Independent, Training Needed or Support Needed
  4.  High Performer Group (identified by each data source)

   Mutli-Variate Analyses

Logistical regression analyses were conducted using the following dependent variables.

  1. Goal Domain Type: Physical Health or Other
  2. Goal Skill Content: Indicated or Not Indicated
  3. Goal Communication Content: Indicated or Not indicated

Key Correlations Pertaining to Health

Chi Square Analyses: Physical Health (n=566) vs Other Goal Types (n=1096)

Physical health goals were more directly correlated with being present in the community (p value <.0001) but negatively correlated with presence of communication content (p value < .0001).

Discussion: The results are generally consistent with the well-established challenges that health providers of all types face in effectively interacting with many patients who have intellectual disabilities. A recent meeting of the Family Medicine Education Consortium focus group on primary care for persons with developmental disabilities (Project Director is a member) identified provider training as the highest need for improving primary care practice and a North Carolina–based group will be developing a curriculum for PCP education over the next two years (Hersey, PA, October, 2010).  

Goals that addressed physical health and personal development were more likely to involve skill learning, compared with other goal domains.
See discussion in next sections for more detail.

Chi Square Analyses: Physical Health Sub-Domains (n = 571)

Sub-Domain Categories: Fitness (n=153), Hygiene (n=37), Medical Care (n=137), Nutrition (n=98), Personal Safety (n=99), Other (n=47)

10 variables were identified to have significant differences (<.0001).

Medical care and fitness were directly correlated with being present in the community, while all categories were negatively correlated with communication content.

Other key sub-domain results follow.

The younger subjects grouping (age 18-39) presented higher rates of participation in fitness, medical care, and nutrition activity.
Goals involving fitness, medical care, and nutrition were rated more highly for goal attainment than other physical health sub-domains.
Subjects assessed as higher functioning were more likely to have goals involving fitness, medical care and nutrition.
Self-guardians were less likely to have goal activity in every sub-domain, compared to others.
Goals involving fitness, hygiene, nutrition and personal safety were more likely to include skill development.

Discussion: The analysis permitted a more detailed look at specific categories pertaining to physical health by supporting and expanding on the findings of the other chi square and regression analyses involving the broader domain categories. The decreased goal activity in self-guardians may be due to those individuals having attained a level of independence where skill development and support are not priorities for service plan goals. Other relationships are discussed in the next section. 

Logistic Regression: Physical Health (n=482) vs Other Goal Types (n=982)

The analyses produced results indicating that physical Health goals were 30 % more likely to include skill development and 70% less likely to have communication content. Physical health goals were 1.6% less likely to be present with each additional year a subject ages.

Discussion: The regression results for relationship with communication content were consistent with the chi square analysis. The direct relationship with skill development was consistent with the analysis of physical health sub-domains and interesting in that it suggests that the tracking of gains is more likely to occur with physical health goals, which might logically include targets such as losing weight (nutrition, fitness), increasing endurance (fitness) or performing hygiene with greater independence. In addition, the Maine system has undertaken initiatives to increase recipient awareness and participation in leading healthy lives, which may have contributed to this finding.  Of the variables resulting in direct correlations with goal attainment (fitness, nutrition and medical care), two correlated directly with skill development. Further study could explore the interrelationship between skill development and goal attainment. The reduced likelihood of physical health goals as subjects age could reflect reduced emphasis on self-management of health in later years of life. 

Other Correlations of Interest

Skill Content &  Communication Content & Goal Attainment

All analyses showed a direct correlation between the presence of communication content and skill development content in goal statements. Regression analyses found that these two characteristics were 2.2 X more likely to be present together, one of the two highest ratios reported. The analyses found a strong relationship between skill development and goals involving more than one-time events (6.6 X) which was is logical given that skill development involves continuous work.

Higher goal attainment was directly correlated with receptive communication ability (p value <.004) and with the mid-ranges (moderate and severe) of intellectual disability (p value < .02). Perhaps worth mentioning was the less significant direct relationship between goal attainment and communication content (p value < .09), in light of the previously stated receptive communication relationship.

Discussion:  Maine OACPD regulations specify expectations for habilitation (skill development), however the importance of communication using non-traditional methods that address individual communication limitations has only begun to be recognized in the field of developmental disability services. Given that analysis for assessment of expressive and receptive communication ability (BMS 99 assessment forms) identified 45% of all subjects as having either moderate or very limited communication limitations, there appears to exist significant unmet need in this area. Further analyses of all sub-domains could provide more detailed insight into the inter-relationships among goal attainment, communication content, skill development present, and receptive communication ability. Considered in total, these findings suggest that further investigation is warranted to distinguish between: intentional and unintended communication, and traditional and non-traditional communication activity, including the impact of each one on outcomes (goal attainment). 

Independence and Skill Development

Generally, characteristics of independence (self-guardianship, higher intellectual dx, less disability, greater expressive and receptive communication ability, independent health maintenance) were negatively correlated with the presence of skill development.

Discussion: More independent individuals may be living more satisfied lives and have less need or desire for learning functional and life skills. Services for these individuals typically involve between 5-25 hours of support per week in order to maintain living situations and receive transportation to needed community resources. That contrasts with higher need individuals receiving high levels of support who have more goals, many which feature targeting skill development (and goal attainment as noted in the section above).  

High Performing Subjects

Each sample source was asked to identify up to 10 subjects who would be considered high performers as described by project staff. This variable exerted greater weight on samples from the OACPD regional offices (50% of each), less than the Mobius and Progress Center samples (25% of each), the combined effect weighting the entire study group towards higher achievement. This additional measure was included in the study to see if correlations of interest could be identified to help explain higher performance.

Chi square analysis of the high performer variable resulted in significant differences for five variables.
Subjects identified as high performing correlated directly with being:  their own guardians (p value < .0001); assessed as having greater expressive and receptive communication ability (p values < .0001); and assessed as higher functioning (p value < .0001).  They were somewhat less likely to be present and participating in the community and more likely to be of one-time duration (p value <.01).

Discussion: Aside from community participation, other correlations appeared to be as expected.  The negative correlation with community participation may be due to the fact that related community-skills are already acquired in high performance subjects, and therefore not the subject of service plan goals.  The correlation with one-time events would be consistent with the explanation provided in the sections above regarding lower levels of skill development with greater independence.  
  

MeHAF Recommended Chi Square Analyses

Guardianship with Disability Level. A significant difference was identified in this analysis (p<.0001). Subjects identified as being their own guardians were directly correlated with being higher functioning.

Health Maintenance Ability with Guardianship and Disability Level. Significant differences were identified in these analyses (p<.0001). Subjects rated as independent in health maintenance were directly correlated with being their own guardians and with being higher functioning.

Medication Administration Ability with Guardianship and Disability Level. Significant differences were identified in these analyses (p<.0001). Subjects who were rated as independent or needing skill training for medication administration were more likely to be their own guardians and to be assessed as higher functioning, compared with subjects requiring support administering medications. 

Goal Domain-Type with Disability Level.  No significant differences were found in the analysis of disability level with domain. However, as reported above, the analysis using physical health sub-domains did find significant differences with disability level, with individuals assessed as higher functioning being more likely to have goals involving fitness, medical care and nutrition.

Discussion: Generally, these results were as expected, higher functioning aligning with increased independence in areas studied. As such, these and many other anticipated findings, generally serve to validate the data collection and analysis methods employed within the project. In medication administration, the finding that the two categories of `independent’ and `skill training needed’ were both more likely to be their own guardians could merit further study.

Additional analyses found health maintenance ability directly correlated with expressive and receptive communication ability and medication administration ability directly correlated with goal attainment. 

Project Limitations

Study limitations include at least the following elements.

The exclusion of other variables influential to outcomes due to the limited available data.
Service plan goals represent only parts of an individual’s life, albeit  important ones.
Secondary data was utilized, which may have been subject to ecological variations that influence results.
Although goal statements classified numbered 1700, statements were taken from service plans for 133 individuals.
A randomized sample selection was not possible, given limited resources available to the project and challenges of collecting data from different sources.

Meeting Goals

The MeHAF/DHHS study `Identifying Health Goals in Service Plans’ project was designed to be exploratory in nature, as an initial inquiry into the content of service plans, as it relates to physical health and other characteristics, and as an opportunity for basic analyses examining variable relationships. These purposes were achieved. In addition, the Director of Quality, DHHS, has indicated interest in supporting further analyses of the data to gain greater detail about variable inter-relationships. 

The project produced several outcomes that can be useful to future data collection and analyses of a developmental disabilities service system activity.

  1. The numbers of variables of significance and the many logical results support the position that classification of goal statements according to different characteristics can be a viable means to collect descriptive information about individual’s lives and service activity.
  2. The project increased understanding about levels and types of service activity underway within the sample studied. Such information can be useful to identify baselines for establishing additional future variables with eventual establishment of milestones reflecting system performance priorities.
  3. Goal attainment can provide one picture of outcomes of service activity.  

MeHAF Role

In addition to the critical role of direct funding and leveraging other support, MeHAF staff participated in the review of initial data and identification of additional areas for analyses (see MeHAF Recommended Chi Square Analyses above).

Next Steps

Analyses and discussion has resulted in identification of other areas for study, dependent on available resources. They include:

Conducting similar study that addresses study limitations by 1) using a larger sample size, 2) employing randomized selection from the entire state population of service recipients, and 3) identifying other data sources (EIS). This is currently being explored with the OACPD.
Identification of new goal characteristics variables for classification, such as classifications pertaining to behavior management, independent living and community functioning.
Refining existing variable classifications to increase reliability and usefulness (ex goal quality).
Use of the study data set for further analysis by the DHHS Office of Quality Improvement (under discussion).

 

For further information about assessing goal attainment at Mobius, contact our director at: webmaster@mobiusinc.org

 

Send mail to webmaster@mobiusinc.org with questions or comments about this web site.
Copyright © 2008-2011 Mobius Incorporated