Case

Subject

Input

DistEA IS (precision)

Manual ground-truth IS

Covered set size

local

remote

all

local

remote

all

local

remote

all

1

MultiChat

Integration test

1 (100%)

13 (69.2%)

14 (71.4%)

1

9

10

3

21

24

2

MultiChat

Integration test

13 (76.9%)

2 (50%)

15 (73.3%)

10

1

11

22

3

25

3

Voldemort

System test

4 (100%)

23 (56.5%)

27 (62.9%)

4

13

17

740

809

1250

4

Voldemort

System test

3 (33.3%)

0 (n/a)

3 (33.3%)

1

0

1

811

440

1250

5

Voldemort

Load test

13 (46.1%)

41 (41.4%)

48 (43.7%)

6

17

21

288

500

1359

average IS

6.8

15.8

21.4

4.7

8

12.0

372.8

354.6

781.6

average precision

71.2%

51.7%

56.9%

subject

MultiChat

input

Integration test

query

<ChatClient.core.Sender: void run()>

DistEA impact set

[local impact Set ] size = 1

<ChatClient.core.Sender: void run()>)>

[remote impact set ] size = 13

<ChatServer.handler.ClientInfo: java.net.Socket getSocket()>

<ChatServer.handler.ClientInfo: ChatServer.handler.ClientListener getListener()>

<ChatServer.handler.ClientInfo: ChatServer.handler.ClientSender getSender()>

<ChatServer.core.ServerDispatcher: void deleteClient(ChatServer.handler.ClientInfo)>

<ChatServer.core.ServerDispatcher: void run()>

<ChatServer.handler.ClientListener: void run()>

<ChatServer.handler.ClientSender: void sendMessage(java.lang.String)>

<ChatServer.core.ServerDispatcher: void dispatchMessage(ChatServer.handler.ClientInfo,java.lang.String)>

<ChatServer.core.ServerDispatcher: void sendMessageToAllClients(java.lang.String)>

<ChatServer.handler.ClientSender: void run()>

<ChatServer.core.ServerDispatcher: java.lang.String getNextMessageFromQueue()>

<ChatServer.handler.ClientSender: java.lang.String getNextMessageFromList()>

<ChatServer.handler.ClientSender: void sendMessageToClient(java.lang.String)>

Manual impact set (as ground truth)

[local impact set] size = 1

<ChatClient.core.Sender: void run()>)>>

Note: the query itself is trivially impacted.

[remote impact set] size = 9

<ChatServer.core.ServerDispatcher: void run()>

<ChatServer.handler.ClientListener: void run()>

<ChatServer.handler.ClientSender: void sendMessage(java.lang.String)>

<ChatServer.core.ServerDispatcher: void dispatchMessage(ChatServer.handler.ClientInfo,java.lang.String)>

<ChatServer.core.ServerDispatcher: void sendMessageToAllClients(java.lang.String)>

<ChatServer.handler.ClientSender: void run()>

<ChatServer.core.ServerDispatcher: java.lang.String getNextMessageFromQueue()>

<ChatServer.handler.ClientSender: java.lang.String getNextMessageFromList()>

<ChatServer.handler.ClientSender: void sendMessageToClient(java.lang.String)>()>

Note: the query prepares messages to be sent in the client, which are later accessed by the listed methods executed in the server process, thus these methods are potentially to be impacted if changes happen to the query. Other four methods executed after the query won’t be impacted as they simply retrieved embedded objects (sender, socket, listener, etc.) for establishing and maintaining the connection with clients.

subject

MultiChat

input

Integration test

query

<ChatServer.core.ServerDispatcher: void dispatchMessage(ChatServer.handler.ClientInfo,java.lang.String)>

DistEA impact set

[local impact Set ] size = 13

<ChatServer.handler.ClientInfo: java.net.Socket getSocket()>

<ChatServer.handler.ClientInfo: ChatServer.handler.ClientListener getListener()>

<ChatServer.handler.ClientInfo: ChatServer.handler.ClientSender getSender()>

<ChatServer.core.ServerDispatcher: void deleteClient(ChatServer.handler.ClientInfo)>

<ChatServer.core.ServerDispatcher: void run()>

<ChatServer.handler.ClientListener: void run()>

<ChatServer.handler.ClientSender: void sendMessage(java.lang.String)>

<ChatServer.core.ServerDispatcher: void dispatchMessage(ChatServer.handler.ClientInfo,java.lang.String)>

<ChatServer.core.ServerDispatcher: void sendMessageToAllClients(java.lang.String)>

<ChatServer.handler.ClientSender: void run()>

<ChatServer.core.ServerDispatcher: java.lang.String getNextMessageFromQueue()>

<ChatServer.handler.ClientSender: java.lang.String getNextMessageFromList()>

<ChatServer.handler.ClientSender: void sendMessageToClient(java.lang.String)>

[remote impact set ] size = 2

<ChatClient.core.Sender: void run()>

<ChatClient.core.MainClient: void main(java.lang.String[])>

Manual impact set (as ground truth)

[local impact Set ] size = 10

<ChatServer.handler.ClientInfo: ChatServer.handler.ClientSender getSender()>

<ChatServer.core.ServerDispatcher: void run()>

<ChatServer.handler.ClientListener: void run()>

<ChatServer.handler.ClientSender: void sendMessage(java.lang.String)>

<ChatServer.core.ServerDispatcher: void dispatchMessage(ChatServer.handler.ClientInfo,java.lang.String)>

<ChatServer.core.ServerDispatcher: void sendMessageToAllClients(java.lang.String)>

<ChatServer.handler.ClientSender: void run()>

<ChatServer.core.ServerDispatcher: java.lang.String getNextMessageFromQueue()>

<ChatServer.handler.ClientSender: java.lang.String getNextMessageFromList()>

<ChatServer.handler.ClientSender: void sendMessageToClient(java.lang.String)>

Note: these methods directly or indirectly manipulate the message to be sent by server to the client; the message is first dispatched to the sender (ClientSender) before the sender actually sends the message out to the current client in the session. The other three methods reported by DistEA won’t be affected by changes to the manipulations in the dispatchMessage method (the query) because they deal with connection maintenance only.

[remote impact set ] size = 1

<ChatClient.core.MainClient: void main(java.lang.String[]

Note: message sent by server is accessed in this main method; the other one reported by DistEA, the run() method in the Sender class, only accesses messages to sent TO the server before the server responds (by sending the message back to client, which is the message prepared in the dispatchMessage method in the server component. So the run() method won’t be impacted.

subject

Voldemort

input

system test

query

<voldemort.client.DefaultStoreClient: voldemort.versioning.Versioned getItemOrThrow(java.lang.Object,voldemort.versioning.Versioned,java.util.List)>

DistEA impact set

[local impact Set ] size = 4

<voldemort.TestClientShutdown: void main(java.lang.String[])>

<voldemort.client.DefaultStoreClient: voldemort.versioning.Versioned get(java.lang.Object,voldemort.versioning.Versioned)>

<voldemort.client.DefaultStoreClient: voldemort.versioning.Versioned getItemOrThrow(java.lang.Object,voldemort.versioning.Versioned,java.util.List)>

<voldemort.client.DefaultStoreClient: voldemort.versioning.Versioned get(java.lang.Object)>

[remote impact set ] size = 23

<voldemort.common.service.AbstractService: void stop()>

<voldemort.cluster.Node: int getId()>

<voldemort.store.stats.Histogram: void resetIfNeeded()>

<voldemort.common.service.AbstractService: boolean isStarted()>

<voldemort.common.service.AbstractService: voldemort.common.service.ServiceType getType()>

<voldemort.server.niosocket.NioSelectorManager: void processEvents()>

<voldemort.cluster.Cluster: java.util.Collection getNodes()>

<voldemort.cluster.failuredetector.FailureDetectorConfig: voldemort.cluster.Cluster getCluster()>

<voldemort.cluster.failuredetector.AbstractFailureDetector: voldemort.cluster.failuredetector.FailureDetectorConfig getConfig()>

<voldemort.store.stats.Histogram: void insert(long)>

<voldemort.cluster.failuredetector.FailureDetectorConfig: voldemort.utils.Time getTime()>

<voldemort.cluster.failuredetector.AsyncRecoveryFailureDetector: void run()>

<voldemort.server.niosocket.NioSelectorManagerStats: void updateSelectStats(int,long,long)>

<voldemort.utils.SystemTime: void sleep(long)>

<voldemort.cluster.failuredetector.AbstractFailureDetector: voldemort.cluster.failuredetector.NodeStatus getNodeStatus(voldemort.cluster.Node)>

<voldemort.store.socket.clientrequest.ClientRequestExecutorFactory$ClientRequestSelectorManager: void processEvents()>

<voldemort.cluster.failuredetector.AbstractFailureDetector: void checkNodeArg(voldemort.cluster.Node)>

<voldemort.common.nio.SelectorManager: void run()>

<voldemort.store.stats.Histogram: void reset()>

<voldemort.common.service.ServiceType: java.lang.String getDisplayName()>

<voldemort.server.VoldemortServer$1: void run()>

<voldemort.cluster.failuredetector.AsyncRecoveryFailureDetector: boolean isAvailable(voldemort.cluster.Node)>

<voldemort.cluster.failuredetector.NodeStatus: boolean isAvailable()>

Manual impact set (as ground truth)

[local impact set] size = 4

<voldemort.client.DefaultStoreClient: voldemort.versioning.Versioned getItemOrThrow(java.lang.Object,voldemort.versioning.Versioned,java.util.List)>

<voldemort.client.DefaultStoreClient: voldemort.versioning.Versioned get(java.lang.Object,voldemort.versioning.Versioned)>

<voldemort.client.DefaultStoreClient: voldemort.versioning.Versioned get(java.lang.Object)>

<voldemort.TestClientShutdown: void main(java.lang.String[])>

Note: the data is retrieved from a Voldemort Store in the query, which returns the data to the other three methods above (except for the query itself), where  the query transitively returned into. Thus, the other three are all to be impacted.

[remote impact] set size = 13

<voldemort.common.nio.SelectorManager: void run()>

<voldemort.server.niosocket.NioSelectorManager: void processEvents()>

<voldemort.cluster.failuredetector.AsyncRecoveryFailureDetector: void run()>

<voldemort.cluster.Node: int getId()>

<voldemort.cluster.failuredetector.AsyncRecoveryFailureDetector: boolean isAvailable(voldemort.cluster.Node)>

<voldemort.cluster.failuredetector.NodeStatus: boolean isAvailable()>

<voldemort.cluster.failuredetector.AbstractFailureDetector: voldemort.cluster.failuredetector.NodeStatus getNodeStatus(voldemort.cluster.Node)>

<voldemort.common.service.AbstractService: voldemort.common.service.ServiceType getType()>

<voldemort.cluster.Cluster: java.util.Collection getNodes()>

<voldemort.store.stats.Histogram: void insert(long)>

<voldemort.server.niosocket.NioSelectorManagerStats: void updateSelectStats(int,long,long)>

<voldemort.cluster.failuredetector.AbstractFailureDetector: void checkNodeArg(voldemort.cluster.Node)>

<voldemort.store.socket.clientrequest.ClientRequestExecutorFactory$ClientRequestSelectorManager: void processEvents()>

Note: the above 13 methods directly or transitively assessed data communicated between the client and voldemort master processes and the data are dependent on the one retrieved from the Store in the query method. The other ten methods executed after the query work for connecting to the master and serving for routine data fault checking and detection, irrelevant of the specific data involved in the communications between the client and the master.

subject

Voldemort

input

system test

query

<voldemort.common.service.AbstractService: void stop()>

DistEA impact set

[local impact Set ] size = 3

<voldemort.common.service.AbstractService: void stop()>

<voldemort.common.service.AbstractService: voldemort.common.service.ServiceType getType()>

<voldemort.common.service.ServiceType: java.lang.String getDisplayName()>

[remote impact set ] size = 0

Manual impact set (as ground truth)

[local impact Set ] size = 1

<voldemort.common.service.AbstractService: void stop()>

Note: besides the query itself is to be trivially impacted, the other two methods produced by DistEA as potentially impacted actually won’t be impacted, because although they are invoked by the query, they have neither data or control dependence on the query in the local process.

[remote impact set] size = 0

Note: the query method executes after the client has disconnected, thus no methods in the client (remote) process for this input can be impacted.

subject

Voldemort

input

load test

query

<voldemort.store.ErrorCodeMapper: short getCode(voldemort.VoldemortException)>

DistEA impact set

[local impact Set ] size = 13

<voldemort.store.ErrorCodeMapper: short getCode(voldemort.VoldemortException)>

<voldemort.common.nio.ByteBufferBackedOutputStream: void expandIfNeeded(int)>

<voldemort.server.niosocket.AsyncRequestHandler: void write(java.nio.channels.SelectionKey)>

<voldemort.common.nio.SelectorManagerWorker: void prepForWrite(java.nio.channels.SelectionKey)>

<voldemort.common.nio.ByteBufferBackedOutputStream: void write(int)>

<voldemort.common.nio.ByteBufferBackedOutputStream: java.nio.ByteBuffer getBuffer()>

<voldemort.common.nio.ByteBufferBackedOutputStream: void write(byte[],int,int)>

<voldemort.common.nio.ByteBufferBackedInputStream: java.nio.ByteBuffer getBuffer()>

<voldemort.server.niosocket.NioSelectorManagerStats: void updateSelectStats(int,long,long)>

<voldemort.store.stats.Histogram: void insert(long)>

<voldemort.server.niosocket.NioSelectorManager: void processEvents()>

<voldemort.store.stats.Histogram: void resetIfNeeded()>

<voldemort.common.nio.SelectorManagerWorker: void run()>

[remote impact set ] size = 41

<voldemort.server.niosocket.NioSelectorManagerStats: void removeConnection()>

<voldemort.server.niosocket.AsyncRequestHandler: void close()>

<voldemort.common.nio.ByteBufferBackedInputStream: void <init>(java.nio.ByteBuffer)>

<voldemort.server.protocol.vold.VoldemortNativeRequestHandler: boolean isCompleteRequest(java.nio.ByteBuffer)>

<voldemort.common.nio.SelectorManagerWorker: void closeInternal()>

<voldemort.common.nio.ByteBufferBackedInputStream: int read()>

<voldemort.common.nio.ByteBufferBackedOutputStream: void close()>

<voldemort.common.nio.ByteBufferBackedInputStream: void close()>

<voldemort.common.nio.SelectorManagerWorker: void handleIncompleteRequest(int)>

<voldemort.server.niosocket.AsyncRequestHandler: void read(java.nio.channels.SelectionKey)>

<voldemort.common.nio.ByteBufferBackedInputStream: int read(byte[],int,int)>

<voldemort.versioning.ObsoleteVersionException: void <init>(java.lang.String)>

<voldemort.store.socket.clientrequest.ClientRequestExecutor: void read(java.nio.channels.SelectionKey)>

<voldemort.store.socket.clientrequest.BlockingClientRequest: boolean isCompleteResponse(java.nio.ByteBuffer)>

<voldemort.store.socket.clientrequest.ClientRequestExecutor: voldemort.store.socket.clientrequest.ClientRequest atomicNullOutClientRequest()>

<voldemort.VoldemortApplicationException: void <init>(java.lang.String)>

<voldemort.server.niosocket.NioSelectorManager: void processEvents()>

<voldemort.common.nio.SelectorManagerWorker: void run()>

<voldemort.common.nio.ByteBufferBackedInputStream: java.nio.ByteBuffer getBuffer()>

<voldemort.server.niosocket.NioSelectorManagerStats: void updateSelectStats(int,long,long)>

<voldemort.store.stats.Histogram: void insert(long)>

<voldemort.server.niosocket.NioSelectorManager: void processEvents()>

<voldemort.store.stats.Histogram: void resetIfNeeded()>

<voldemort.common.nio.SelectorManagerWorker: void run()>

<voldemort.store.ErrorCodeMapper: voldemort.VoldemortException getError(short,java.lang.String)>

<voldemort.store.socket.clientrequest.BlockingClientRequest: void parseResponse(java.io.DataInputStream)>

<voldemort.store.socket.clientrequest.AbstractClientRequest: void complete()>

<voldemort.store.socket.clientrequest.BlockingClientRequest: void complete()>

<voldemort.store.socket.clientrequest.ClientRequestExecutor: voldemort.store.socket.clientrequest.ClientRequest completeClientRequest()>

<voldemort.store.socket.clientrequest.PutClientRequest: java.lang.Object parseResponseInternal(java.io.DataInputStream)>

<voldemort.utils.ReflectUtils: java.lang.Object callConstructor(java.lang.Class,java.lang.Class[],java.lang.Object[])>

<voldemort.VoldemortException: void <init>(java.lang.String)>

<voldemort.store.socket.clientrequest.AbstractClientRequest: void parseResponse(java.io.DataInputStream)>

<voldemort.utils.ReflectUtils: java.lang.Object callConstructor(java.lang.Class,java.lang.Object[])>

<voldemort.store.socket.clientrequest.PutClientRequest: boolean isCompleteResponse(java.nio.ByteBuffer)>

<voldemort.client.protocol.vold.VoldemortNativeClientRequestFormat: void checkException(java.io.DataInputStream)>

<voldemort.versioning.ObsoleteVersionException: java.lang.Throwable fillInStackTrace()>

<voldemort.store.socket.clientrequest.ClientRequestExecutorFactory$ClientRequestSelectorManager: void processEvents()>

<voldemort.client.protocol.vold.VoldemortNativeClientRequestFormat: boolean isCompleteResponse(java.nio.ByteBuffer,byte)>

<voldemort.store.socket.clientrequest.PutClientRequest: java.lang.Void parseResponseInternal(java.io.DataInputStream)>

<voldemort.client.protocol.vold.VoldemortNativeClientRequestFormat: boolean isCompletePutResponse(java.nio.ByteBuffer)>

<voldemort.client.protocol.vold.VoldemortNativeClientRequestFormat: void readPutResponse(java.io.DataInputStream)>

<voldemort.store.socket.clientrequest.ClientRequestExecutor: boolean checkTimeout()>

Manual impact set (as ground truth)

[local impact Set ] size = 6

<voldemort.store.ErrorCodeMapper: short getCode(voldemort.VoldemortException)>

<voldemort.server.niosocket.AsyncRequestHandler: void write(java.nio.channels.SelectionKey)>

<voldemort.common.nio.ByteBufferBackedOutputStream: void write(int)>

<voldemort.common.nio.ByteBufferBackedOutputStream: void write(byte[],int,int)>

<voldemort.server.niosocket.NioSelectorManager: void processEvents()>

<voldemort.common.nio.SelectorManagerWorker: void run()>

Note: these six methods directly (in the query itself) or indirectly used the Error Code returned by the query thus they will be impacted if developers make a change to the query. The other seven methods reported never involve error handling tasks and they did not use any such error code produced by the query; And the query is a simply method just returning numeric code mapped to by an exception type (class), it did not control the execution of any other methods.

[remote impact set ] size = 17

<voldemort.common.nio.ByteBufferBackedInputStream: int read()>

<voldemort.server.niosocket.AsyncRequestHandler: void read(java.nio.channels.SelectionKey)>

<voldemort.common.nio.ByteBufferBackedInputStream: int read(byte[],int,int)>

<voldemort.store.socket.clientrequest.ClientRequestExecutor: void read(java.nio.channels.SelectionKey)>

<voldemort.store.ErrorCodeMapper: voldemort.VoldemortException getError(short,java.lang.String)>

<voldemort.store.socket.clientrequest.BlockingClientRequest: void parseResponse(java.io.DataInputStream)>

<voldemort.store.socket.clientrequest.ClientRequestExecutor: voldemort.store.socket.clientrequest.ClientRequest completeClientRequest()>

<voldemort.store.socket.clientrequest.PutClientRequest: java.lang.Object parseResponseInternal(java.io.DataInputStream)>

<voldemort.utils.ReflectUtils: java.lang.Object callConstructor(java.lang.Class,java.lang.Class[],java.lang.Object[])>

<voldemort.utils.ReflectUtils: java.lang.Object callConstructor(java.lang.Class,java.lang.Object[])>

<voldemort.client.protocol.vold.VoldemortNativeClientRequestFormat: void checkException(java.io.DataInputStream)>

<voldemort.versioning.ObsoleteVersionException: java.lang.Throwable fillInStackTrace()>

<voldemort.store.socket.clientrequest.ClientRequestExecutorFactory$ClientRequestSelectorManager: void processEvents()>

<voldemort.store.socket.clientrequest.PutClientRequest: java.lang.Void parseResponseInternal(java.io.DataInputStream)>

<voldemort.client.protocol.vold.VoldemortNativeClientRequestFormat: void readPutResponse(java.io.DataInputStream)>

<voldemort.server.niosocket.NioSelectorManager: void processEvents()>

<voldemort.common.nio.SelectorManagerWorker: void run()>

Note: This query is defined (and executed of course) in a common module used by both the voldemort master and its clients.  DistEA reported 6 common impacts between local and remote impact sets, among which only two can be potentially impacted (<voldemort.server.niosocket.NioSelectorManager: void processEvents()>, and <voldemort.common.nio.SelectorManagerWorker: void run()>). These two methods are rightly included by DistEA in this remote impact set (for the same reason as they were included in the local impact set). Other ground-truth impacts, although did not manipulate the error code directly, determined  whether and/or which exceptions were triggered based on the error code returned by the query, after which error codes for those exceptions will be retrieved again after executing these methods in the remote process; thus, changes to this query can affect the above 17 methods. Other 24 methods reported by DistEA in its remote impact set did not use the error code, of which most were serving for maintenance (heartbeat) purposes.