Notice: Please cite the following paper if you are using TaintBench:

@article{TaintBench2021,
author = {Linghui Luo and Felix Pauck and Goran Piskachev and Manuel Benz and Ivan Pashchenko and Martin Mory and Eric Bodden and Ben Hermann and Fabio Massacci},
title = {TaintBench: Automatic Real-World Malware Benchmarking of Android Taint Analyses},
year = {2021},
journal = {Empirical Software Engineering},
note = {to appear}
}

How does TaintBench compare to DroidBench?

List of Evaluated Benchmark Suites

Usage of Sources and Sinks

usage of sources and sinks

Call-Graph Complexity

  • Results:

call-graph complexity

Code Complexity

  • Results:

code complexity

code complexity

How effective are taint analysis tools on TaintBench compared to DroidBench?

List of Evaluated Tools

Tool Version Source
Amandroid November 2017 (3.1.2) Link
Amandroid* December 2018 (3.2.0) Link
FlowDroid April 2017 (Nightly) Link
FlowDroid* January 2019 (2.7.1) Link

Download All Tools

Experiment 1 (Default)

Configuration: All tools are executed in their default configuration. Sources and sinks configured for the tools can be found below:

Experiment 2 (Suite-level)

Configuration: All tools are configured with sources and sinks defined in benchmark suite. Sources and sinks of the benchmark suites configured for the tools can be found below:

Experiment 3 (App-level)

Configuration: For each benchmark app, a list of sources and sinks defined in this app is used to configure all tools. Each tool analyzes each benchmark app with the associated list of sources and sinks.

Experiment 4 (Case-level)

Configuration: For each benchmark case (taint flow), only the source and sink defined in this case is used to configure all tools.

Assume a benchmark app contains N benchmark cases, each tool analyzes the benchmark app N times. Each time the tools are configured with the associated source and sink for the respective benchmark case.

Experiment 5 (Minified App)

For each positive benchmark case, a minified Apk is generated by the MinApkGenerator.

Experiment 6 (Delta App)

For each positive benchmark case, a delta Apk is generated by the DeltaApkGenerator.