Results of Manual Inspection of the Ground Truth in the Predictive Accuracy Study of Dynamic Impact Analysis

NanoXML:

[single method change]

change location (statement no.)

3582

query (method)

net.n3.nanoxml.XMLElement: boolean isLeaf()

tests

all covering tests: 74, 75, 76

randomly selected: 74, 75, 76

PI/EAS results

<net.n3.nanoxml.XMLElement: boolean isLeaf()>

<CheckLeaf1_wy_v1: void main(java.lang.String[])>

<AllDriversMain: void main(java.lang.String[])>

<CheckLeaf2_wy_v1: void main(java.lang.String[])>

<CheckLeaf3_wy_v1: void main(java.lang.String[])>

mDEA results

<net.n3.nanoxml.XMLElement: boolean isLeaf()>

<CheckLeaf1_wy_v1: void main(java.lang.String[])>

<CheckLeaf2_wy_v1: void main(java.lang.String[])>

<CheckLeaf3_wy_v1: void main(java.lang.String[])>

manual ground-truth

<net.n3.nanoxml.XMLElement: boolean isLeaf()>

<CheckLeaf1_wy_v1: void main(java.lang.String[])>

<CheckLeaf2_wy_v1: void main(java.lang.String[])>

<CheckLeaf3_wy_v1: void main(java.lang.String[])>

ground-truth conformance

Fully conformed:

The changed method, although transitively returned into <AllDriversMain: void main(java.lang.String[])>, which is thus identified as impacted by PI/EAS, did not affect (any execution states in) this test driver entry method

The change is for a boolean value returned by the changed method; when this value is changed, only its direct caller and itself are affected because such change did not propagate further along

change location (statement no.)

4068

query (method)

method: net.n3.nanoxml.XMLUtil: void errorWrongClosingTag(net.n3.nanoxml.IXMLReader,java.lang.String,java.lang.String)

tests

all covering test: 128, 129, 145, 149, 153, 157, 161, 165, 169, 173, 185, 189

randomly selected: 129, 153, 185

PI/EAS results

<net.n3.nanoxml.StdXMLParser: java.lang.Object parse()>

<Parser1_vw_v1: void main(java.lang.String[])>

<net.n3.nanoxml.StdXMLParser: void processElement()>

<net.n3.nanoxml.XMLUtil: void errorWrongClosingTag(net.n3.nanoxml.IXMLReader,java.lang.String,java.lang.String)>

<net.n3.nanoxml.StdXMLParser: void scanData()>

<AllDriversMain: void main(java.lang.String[])>

<net.n3.nanoxml.StdXMLReader: int getLineNr()>

<net.n3.nanoxml.StdXMLParser: void scanSomeTag(boolean)>

<net.n3.nanoxml.XMLParseException: void <init>(int,java.lang.String)>

mDEA results as ground-truth

<net.n3.nanoxml.XMLParseException: void <init>(int,java.lang.String)>

<net.n3.nanoxml.XMLUtil: void errorWrongClosingTag(net.n3.nanoxml.IXMLReader,java.lang.String,java.lang.String)>

manual ground-truth

<net.n3.nanoxml.XMLParseException: void <init>(int,java.lang.String)>

<net.n3.nanoxml.XMLUtil: void errorWrongClosingTag(net.n3.nanoxml.IXMLReader,java.lang.String,java.lang.String)>

ground-truth conformance

Fully conformed:

The change involves the construction of an exception message; when this change is made, only the enclosing method is affected, plus the constructor of the exception class which takes the message as the single parameter.

[multi-method change]

change locations (statement no.s)

4031

1007

queries (methods)

<net.n3.nanoxml.XMLUtil: void errorExpectedInput(net.n3.nanoxml.IXMLReader,java.lang.String)>

<Parser6_vw_v0: void main(java.lang.String[])>

tests

all covering tests: 133,134, 136, 141

randomly selected: 133,136,141

PI/EAS results

<net.n3.nanoxml.StdXMLParser: java.lang.Object parse()>

<net.n3.nanoxml.XMLParseException: void <init>(int,java.lang.String)>

<net.n3.nanoxml.XMLUtil: void errorExpectedInput(net.n3.nanoxml.IXMLReader,java.lang.String)>

<Parser1_vw_v1: void main(java.lang.String[])>

<net.n3.nanoxml.StdXMLParser: void processElement()>

<net.n3.nanoxml.XMLUtil: java.lang.String scanString(net.n3.nanoxml.IXMLReader,char,boolean,net.n3.nanoxml.XMLEntityResolver)>

<net.n3.nanoxml.StdXMLParser: void scanData()>

<AllDriversMain: void main(java.lang.String[])>

<net.n3.nanoxml.StdXMLReader: int getLineNr()>

<net.n3.nanoxml.StdXMLParser: void processAttribute()>

<net.n3.nanoxml.StdXMLParser: void scanSomeTag(boolean)>

<net.n3.nanoxml.StdXMLParser: void processSpecialTag(boolean)>

<net.n3.nanoxml.StdXMLParser: void processDocType()>

mDEA results as ground-truth

<net.n3.nanoxml.XMLParseException: void <init>(int,java.lang.String)>

<net.n3.nanoxml.XMLUtil: void errorExpectedInput(net.n3.nanoxml.IXMLReader,java.lang.String)>

<Parser6_vw_v0: void main(java.lang.String[])>

manual ground-truth

<net.n3.nanoxml.XMLParseException: void <init>(int,java.lang.String)>

<net.n3.nanoxml.XMLUtil: void errorExpectedInput(net.n3.nanoxml.IXMLReader,java.lang.String)>

<Parser6_vw_v0: void main(java.lang.String[])>

ground-truth conformance

Fully conformed:

Only the two queried methods themselves are trivially impacted because of the changes made in them; the only other truly affected one is the exception class constructor, which takes the changed message as its sole input parameter.  Because the value change in the second method did not change the predicate anyway, and the second change only affects the exception message, which was not further used by method executed after, only these three methods identified manually above were truly impacted, which is consistent with the mDEA-based ground truth used in our study.

Schedule1:

[single method change]

change location (statement no.)

317

query (method)

<ScheduleClass: void upgrade_process_prio(int,float)>

tests

all covering tests: 2647 of the total 2650 tests all covered the change

randomly selected: 101, 895, 1793

PI/EAS results

<ScheduleClass: void finish_process()>

<ScheduleClass: void block_process()>

<ScheduleClass: ScheduleClass$Ele new_process(int)>

<ScheduleClass: ScheduleClass$List append_ele(ScheduleClass$List,ScheduleClass$Ele)>

<ScheduleClass: void schedule()>

<ScheduleClass: ScheduleClass$Ele new_ele(int)>

<ScheduleClass: void upgrade_process_prio(int,float)>

<ScheduleClass: void add_process(int)>

<ScheduleClass: void quantum_expire()>

<ScheduleClass: void _main(java.lang.String[])>

<ScheduleClass: void finish_all_processes()>

<ScheduleClass: void main(java.lang.String[])>

<ScheduleClass$Ele: void <init>()>

<ScheduleClass: ScheduleClass$List del_ele(ScheduleClass$List,ScheduleClass$Ele)>

<ScheduleClass: void unblock_process(float)>

<ScheduleClass$List: void <init>()>

<ScheduleClass: ScheduleClass$List new_list()>

<ScheduleClass: ScheduleClass$Ele find_nth(ScheduleClass$List,int)>

mDEA results as ground-truth

<ScheduleClass: void upgrade_process_prio(int,float)>

<ScheduleClass: ScheduleClass$Ele find_nth(ScheduleClass$List,int)>

<ScheduleClass: ScheduleClass$List del_ele(ScheduleClass$List,ScheduleClass$Ele)>

<ScheduleClass: void finish_process()>

<ScheduleClass: ScheduleClass$List append_ele(ScheduleClass$List,ScheduleClass$Ele)>

<ScheduleClass: void schedule()>

<ScheduleClass: void unblock_process(float)>

<ScheduleClass: void quantum_expire()>

manual ground-truth

<ScheduleClass: void upgrade_process_prio(int,float)>

<ScheduleClass: ScheduleClass$Ele find_nth(ScheduleClass$List,int)>

<ScheduleClass: ScheduleClass$List del_ele(ScheduleClass$List,ScheduleClass$Ele)>

<ScheduleClass: ScheduleClass$List append_ele(ScheduleClass$List,ScheduleClass$Ele)>

<ScheduleClass: void schedule()>

<ScheduleClass: void unblock_process(float)>

<ScheduleClass: void finish_process()>

<ScheduleClass: void quantum_expire()>

ground-truth conformance

Fully conformed:

All the methods identified by PI/EAS were executed after the queried one, among which only those listed in the

manual ground truth were affected by the change because others did not use variables or predicates

directly or transitively dependent on the changed variable.

change location (statement no.)

210

query (method)

<ScheduleClass: void finish_all_processes()>

tests

all covering tests: 2433 out of the total 2650 tests covered the change

randomly selected: 33, 126, 2223

PI/EAS results

<ScheduleClass: void finish_process()>

<ScheduleClass: void unblock_process(float)>

<ScheduleClass: void block_process()>

<ScheduleClass: ScheduleClass$Ele new_process(int)>

<ScheduleClass: ScheduleClass$List append_ele(ScheduleClass$List,ScheduleClass$Ele)>

<ScheduleClass: void schedule()>

<ScheduleClass: ScheduleClass$Ele new_ele(int)>

<ScheduleClass: void upgrade_process_prio(int,float)>

<ScheduleClass: void add_process(int)>

<ScheduleClass: void quantum_expire()>

<ScheduleClass: void _main(java.lang.String[])>

<ScheduleClass: void finish_all_processes()>

<ScheduleClass: void main(java.lang.String[])>

<ScheduleClass: ScheduleClass$Ele find_nth(ScheduleClass$List,int)>

<ScheduleClass$Ele: void <init>()>

<ScheduleClass: ScheduleClass$List del_ele(ScheduleClass$List,ScheduleClass$Ele)>

mDEA results as ground-truth

<ScheduleClass: void finish_process()>

<ScheduleClass: void finish_all_processes()>

<ScheduleClass: void schedule()>

manual ground-truth

<ScheduleClass: void finish_process()>

<ScheduleClass: void finish_all_processes()>

<ScheduleClass: void schedule()>

ground-truth conformance

Fully conformed:

Change at this location affects the number of iterations of a loop;

however, only the three methods are affected through variables changed because of the change in the number of iterations (via a global variable that these methods used).

[multi-method change]

change locations (statement no.s)

291

13

queries (methods)

<ScheduleClass: void unblock_process(float)>

<ScheduleClass: ScheduleClass$Ele find_nth(ScheduleClass$List,int)>

tests

all covering tests: 2109 out of the total 2650 tests covered the change

randomly selected: 579, 1501, 2645

PI/EAS results

<ScheduleClass: void unblock_process(float)>

<ScheduleClass: void _main(java.lang.String[])>

<ScheduleClass: void main(java.lang.String[])>

<ScheduleClass: void finish_process()>

<ScheduleClass: void block_process()>

<ScheduleClass: ScheduleClass$Ele new_process(int)>

<ScheduleClass: ScheduleClass$List append_ele(ScheduleClass$List,ScheduleClass$Ele)>

<ScheduleClass$List: void <init>()>

<ScheduleClass: ScheduleClass$List new_list()>

<ScheduleClass: void schedule()>

<ScheduleClass: ScheduleClass$Ele new_ele(int)>

<ScheduleClass: void add_process(int)>

<ScheduleClass$Ele: void <init>()>

<ScheduleClass: ScheduleClass$List del_ele(ScheduleClass$List,ScheduleClass$Ele)>

<ScheduleClass: void upgrade_process_prio(int,float)>

<ScheduleClass: ScheduleClass$Ele find_nth(ScheduleClass$List,int)>

<ScheduleClass: void finish_all_processes()>

mDEA results as ground-truth

<ScheduleClass$Ele: void <init>()>

<ScheduleClass: ScheduleClass$Ele find_nth(ScheduleClass$List,int)>

<ScheduleClass: ScheduleClass$Ele new_ele(int)>

<ScheduleClass: ScheduleClass$Ele new_process(int)>

<ScheduleClass: ScheduleClass$List append_ele(ScheduleClass$List,ScheduleClass$Ele)>

<ScheduleClass: ScheduleClass$List del_ele(ScheduleClass$List,ScheduleClass$Ele)>

<ScheduleClass: void _main(java.lang.String[])>

<ScheduleClass: void upgrade_process_prio(int,float)>

<ScheduleClass: void unblock_process(float)>

<ScheduleClass: void finish_all_processes()>

<ScheduleClass: void block_process()>

<ScheduleClass: void add_process(int)>

<ScheduleClass: void finish_process()>

<ScheduleClass: void schedule()>

manual ground-truth

<ScheduleClass: void unblock_process(float)>

<ScheduleClass: ScheduleClass$Ele find_nth(ScheduleClass$List,int)>

<ScheduleClass: ScheduleClass$List del_ele(ScheduleClass$List,ScheduleClass$Ele)>

<ScheduleClass: ScheduleClass$List append_ele(ScheduleClass$List,ScheduleClass$Ele)>

<ScheduleClass: void upgrade_process_prio(int,float)>

<ScheduleClass: void finish_process()>

<ScheduleClass: void schedule()>

<ScheduleClass: void add_process(int)>

<ScheduleClass: ScheduleClass$Ele new_process(int)>

<ScheduleClass: ScheduleClass$Ele new_ele(int)>

<ScheduleClass: void finish_all_processes()>

<ScheduleClass: void block_process()>

<ScheduleClass: void unblock_process(float)>

<ScheduleClass: void _main(java.lang.String[])>

ground-truth conformance

Fully conformed:

The three false-positive impacts found by PI/EAS include two initializers that used constants to initialize some global variables, thus they were not affected by the two changes; the other one is the entry method, which was finally returned into, yet it was not really affected by the changes because it does not use any of the two changes directly or transitively.