company logo

Test framework actions (Linux)

The bash test framework consists of a number of procedures for managing tests under Linux.

In order to run a test, all required resources are copied to the work area ( work_area ) which is structured according to the requirements of the component to be tested. When running a series of tests in the test framework, the work area will be overwritten each time, a new test is started. In order to guarantee traceability, the work area has to be copied to the test suite running the test (e.g. as zip file). Here, only differing files are stored to the failed directory in the test suite.

The test framework provides a set of common actions for managing test preparation and evaluation. In addition, each specific test environment has to provide adopted actions named preprocessing , run and postprocessing (bash file names without extension), which are usually stored in main_suite / actions directory, and possibly overloaded in several test cases or test suites. Test framework actions (procedures) are stored in the test root directory.

Several actions that are called internally by the framework are defined as functions in TestSuite.sh . Framework actions that may be called from command line are RunTest.sh , RunMany.sh , SetupWorkArea.sh and TestOut.sh . In order to support parallel testing, the work area location may also be passed as parameter to RunTest or RunMany action.

Starting test run(s)

In order to execute a single or a number test cases, framework actions RunTest or RunMany may be called.

For executing a single test case, RunTest is called. In order to locate the proper test case, the relative path has to be defined like API/Local/Property .

In order to run the complete test suite hierarchy, RunMany may to be called. The procedure executes all test case directories containing a run file. In order to filter test cases for execution, one may rename the run file in test case directories to be excluded.

For each test directory, runSingle will be called. runSingle executes a single test suite after checking the presence of a run file in the directory. When the run file exists, testRun is called.

# RunTest.sh

#!/bin/bash

source ./TestSuite.sh

# execute single test

  # 1 - relative test suite path (e.g. 0/1/2)

  test_suite=$1 # CHAR

  # 2 - test run directory (${RUN_PATH%})

  run_root=$2 # CHAR

  # 3 - main suite directory (${ROOT_PATH})

  main_suite=$3 # CHAR

  # 4 - test root directory ($PWD)

  root_dir=$4 # CHAR

  # 5 - work area location (${WORK_DIR})

  WORK_dir=$4 # CHAR

  settings ${test_suite} ${run_root} ${main_suite} ${root_dir} ${work_dir} ;

  pushd ${RUN_ROOT} > /dev/null

  runSingle ./${test_suite}

  popd > /dev/null

# RunMany.sh

#!/bin/bash

source ./TestSuite.sh

# execute single test

  # 1 - relative test suite path (e.g. 0/1/2)

  test_suite=$1 # CHAR

  # 2 - test run directory (${RUN_PATH%})

  run_root=$2 # CHAR

  # 3 - main suite directory (${ROOT_PATH})

  main_suite=$3 # CHAR

  # 4 - test root directory ($PWD)

  root_dir=$4 # CHAR

  # 5 - work area location (${WORK_DIR})

  work_dir=$4 # CHAR

  settings ${test_suite} ${run_root} ${main_suite} ${root_dir} ${work_dir} ;

  pushd %RUN_ROOT%

  find ./ -type d -exec ${ROOT_DIR}\runSingle.sh {} $1 ;

  popd

# run single test in hierarchy (TestSuite.sh)

runSingle() {

# 1 - test directory below main suite .../main_suite

  test_suite=$1 # CHAR

  #CODE

  if [ -f ${MAIN_SUITE}/$1/run ]; then

    testRun ${test_suite};

  fi

}

Running a test within the frame work

In order to run a test, the testRun action is provided by the test framework ( TestSuite.sh ). The testRun action updates settings for environment variables, prepares the work area and calls test environment specific actions for the selected work suite. settings and prepare are framework actions, which cannot be overloaded, while preprocessing , run and postprocessing are test environment specific actions, which have to be defined for each specific test environment in top or lower test suite or test case actions directory.

After setting test environment variables ( settings action), the prepare action copies test data ( actions , data , expected ) to corresponding directories in work_area from the test suite hierarchy. Test data from test cases and test suites is copied recursively ( copyRecursive ) from the test suite hierarchy by copying resources from the test case to be tested and all its parent test suites. A directory is considered as test suite, when it contains a file suite . Copying stops, when the first directory was found, which is not a test suite. Files existing on lower levels in the hierarchy, always will overwrite files with the same name provided on higher levels.

When NO_RUN has been set to YES ( SetupWorkArea ), the testRun action terminates without executing the test run. Otherwise actions preprocessing , run and postprocessing are called.

Finally the test result is checked by comparing data and writing differences to COMPARE_OUT ( compare.out ) file in test run directory. When COMPARE_OUT is empty, the test status is success , and failed otherwise. The result is written to console and to results.out in the test run directory by the report action.

# selected actions called by testRun in TestSuite.sh

# execute singele test

testRun() {

# 1 - test directory below main suite .../main_suite

  test_suite=$1 # CHAR

  #CODE

  pushd . > /dev/null

  # prepare test work suite ... (global actions)

  settings ${test_suite} ${RUN_ROOT} ${MAIN_SUITE} ${ROOT_DIR}

  prepare

  source ${TEST_ACT}/preprocessing

  if [ ! "${NO_RUN}" == "YES" ]; then

    # run test ... (actions may be overloaded)

    preprocessing ${test_suite}

    ${TEST_ACT}/run ${test_suite}

    ${TEST_ACT}/postprocessing ${test_suite}

    # create test report

    report ${test_suite}

  fi

  popd > /dev/null

}

# Set global test frame work variables

settings() {

  # 1 - relative test suite path (e.g. 0/1/2)

  test_suite=$1 # CHAR

  # 2 - test run directory (${RUN_PATH%})

  run_root=$2 # CHAR

  # 3 - main suite directory (${ROOT_PATH})

  main_suite=$3 # CHAR

  # 4 - test root directory ($PWD)

  root_dir=$4 # CHAR

  # 5 - temporary work directory (${WORK_DIR})

  work_dir=$5 # CHAR

  #CODE

  if [ "${root_dir}" == "" ]; then

    export ROOT_DIR=$PWD;

  else

    export ROOT_DIR=${root_dir};

  fi

    

  if [ "${main_suite}" == "" ]; then

    export MAIN_SUITE=${ROOT_PATH};

  else

    export MAIN_SUITE=${main_suite};

  fi

  if [ "${run_root}" == "" ]; then

    export RUN_ROOT=${RUN_PATH};

  else

    export RUN_ROOT=${run_root};

  fi

  if [ ! -d ${RUN_ROOT} ]; then

    mkdir -p ${RUN_ROOT};

  fi

  if [ "${test_suite}" == "" ]; then

    export TEST_SUITE=${MAIN_SUITE};

  else

    export TEST_SUITE=${MAIN_SUITE}/${test_suite};

  fi

  if [ "${work_dir}" == "" ]; then

    if [ "${WORK_DIR}" == "" ]; then

      WORK_AREA=${ROOT_DIR}/work_area;

else

      WORK_AREA=${WORK_DIR};

fi

  else

    WORK_AREA=${work_dir};

  fi

  if [ ! -d ${WORK_AREA}/actions ]; then

    mkdir -p ${RUN_ROOT}/actions;

  fi

  if [ ! -d ${WORK_AREA}/data ]; then

    mkdir -p ${RUN_ROOT}/data;

  fi

  if [ ! -d ${WORK_AREA}/expected ]; then

    mkdir -p ${RUN_ROOT}/expected;

  fi

  

  export WORK_AREA

  export TEST_ACT=${WORK_AREA}/actions

  export BIN_DIR=${ROOT_DIR}/binaries

  export TOOLS_DIR=${BIN_DIR}/tools

  export TEST_RUN=${RUN_ROOT}/${test_suite}

  

  export LOGFILE_OUT=${RUN_ROOT}/logfile.out

  export RESULTS_OUT=${RUN_ROOT}/results.out

  export REPORT_OUT=${RUN_ROOT}/report.out

  export ERRORS_OUT=${TEST_RUN}/errors.out

  export COMPARE_OUT=${TEST_RUN}/compare.out

  export LD_LIBRARY_PATH=/usr/local/lib

  export ODABA_TOOLS=/usr/local/lib/odaba/tools

  export TEST_NAME=$(basename ${TEST_SUITE})

}

# Initialize test folder

prepare() {

  #CODE

  # delete last test suite results

  rm -f ${COMPARE_OUT}

  rm -f ${ERRORS_OUT}

  # delete work suite data

  rm -rf ${WORK_AREA}/data/*

  rm -rf ${WORK_AREA}/expected/*

  rm -rf ${WORK_AREA}/actions/*

  rm -rf ${TEST_RUN}/failed/*

  pushd ${TEST_SUITE} > /dev/null

  copyRecursive

  popd > /dev/null

}

# Hierarchical copies contents of data, expected and actions into the ${work_area}

copyRecursive() {

  #CODE

  if [ -f suite ]; then

  pushd .. > /dev/null

    copyRecursive

    popd > /dev/null

    cp -r data/* ${WORK_AREA}/data 2>/dev/null

    cp -r expected/* ${WORK_AREA}/expected 2>/dev/null

    cp -r actions/* ${WORK_AREA}/actions 2>/dev/null

  fi

}

# evaluate termination code

report() {

# 1 - test directory below main suite .../main_suite

  test_suite=$1 # CHAR

  test_result=success

  #CODE

  if [ -f ${COMPARE_OUT} ]; then

    diff "${ROOT_DIR}/main_suite/compare.top" "${COMPARE_OUT}" > /dev/null || test_result=failed

  else

    test_result=failed

echo "... nothing to compare (expected empty)" >${COMPARE_OUT}

  fi

  echo "${test_suite}; ${test_result}" >>${RESULTS_OUT}

  echo "${test_suite} terminated: ${test_result}"

  if [ ! -f ${TEST_RUN}/run ]; then

    cp ${TEST_SUITE}/run ${TEST_RUN}/run

  fi

}

Output test protokol

Remarks about the test runs may have been stored in each test run directory in the run file. In order to print out a comprehensive test protocol, one may call TestOut framework action ( TestOut.sh ). The action calls outRun for each sub directory containing a run file.

More reporting features are provided with ODABA Test Browser application.

# TestOut.sh

#!/bin/bash

source ./TestSuite.sh

# execute single test

  # 1 - relative test suite path (e.g. 0/1/2)

  test_suite=$1 # CHAR

  # 2 - test run directory (${RUN_PATH%})

  run_root=$2 # CHAR

  # 3 - main suite directory (${ROOT_PATH})

  main_suite=$3 # CHAR

  # 4 - test root directory ($PWD)

  root_dir=$4 # CHAR

  settings ${test_suite} ${run_root} ${main_suite} ${root_dir} ;

  

  rm -f ${REPORT_OUT}

  pushd ${RUN_ROOT}

  find ./ -type d -exec ${ROOT_DIR}/outRun.sh {} ${test_suite} ;

  type ${RESULTS_OUT}

  popd

# create test run summary and print to REPORT_OUT (TestSuite.sh)

outRun() {

# 1 - test directory below main suite .../main_suite

  test_suite=$1 # CHAR

  #CODE

  settings %1 %ROOT_DIR%

  if [ -f ${TEST_SUITE}/run ]; then

    pushd ${TEST_RUN} > /dev/null

    echo Tested ${test_suite}>>${REPORT_OUT}

    if [ -f run ]; then

  type run >>${REPORT_OUT}

fi

    echo .>>${REPORT_OUT}

    echo ----------------------------------------->>${REPORT_OUT}

    popd > /dev/null

  fi

}