A Framework For Testing Probabilistic Programming Systems
Tutorial 3 : Understanding and Modifying ProbFuzz Config
ProbFuzz has a config.json file which controls what tools to run, which template to use and what tool specific parameters to use. We will explain the various parts next.
Global Configurations
  • max_threads: No. of threads to use for running ProbFuzz. Default is 1
  • structured: Flag to specify whether to use domain knowledge while populating the template. Otherwise, the programs might fail trivially. Default is True
  • output_dir: The directory for storing results. Default is "output"
  • current_template: The template to use for fuzzing
  • metric: The metric to be used for current template. Current options are simple_smape, lr_smape, mlr_smape and lrc_smape. The user can choose to add more metrics for any template that is added, in the metrics/ folder.
  • runConfigurations: The configuration for each tool. Each configuration has the following fields:
    • tool: Name of the tool
    • enabled: Run the tool or disable. Default is True
    • algorithm: Use specific inference algorithm or all
    • timeout: Maximum time to run each program in this tool. Default is 5 minutes
    • python: The python binary to use to run the tool. It can be the default system python or a virtual environment as well
  • templates Map of template names to their relative file locations
Tool specific configurations
  • edward
    • inferences: It has the list of supported inferences and their specific parameters. This should not be edited except for the ig field which if true skips a particular inference
  • pyro
    • optimizers: list of various optimizers and their respective parameters. Users can add some more optimizers from the pyro library if needed. Otherwise, this should not be edited.