Merge branch 'master' of github.com:squaresLab/BugZoo

Author: ChrisTimperley

.gitignore

@@ -1,3 +1,8 @@
+Pipfile
+Pipfile.lock
+bugzood.log
+docs-html
+
 .mypy_cache
 *.pyc
 

CHANGELOG.md

@@ -1,3 +1,32 @@
+## 2.1.7 (2018-05-20)
+
+### Features
+
+* `bugzood` now allows users to log to a given file.
+* Added `**kwargs` to all HTTP methods in low-level API client; allows more
+  information to be forwarded onto `requests`.
+* Added `build` method as an alias of `compile` in both client- and
+  server-side container managers.
+* Added more detailed logging to client file manager.
+* Added `__repr__` to `Spectra`.
+* Added check to `BugZoo` to ensure that the Docker server is responsive.
+* Added slightly better error handling to `mktemp` in server-side container
+  manager.
+* Added better logging and error reporting to `command` in server-side
+  container manager.
+* Added `POST /containers/<uid>/build` endpoint to server.
+
+### Bug Fixes
+
+* Fixed a bug in logging that prevented `DEBUG` level calls from being written
+  to file or passed to the stdout.
+* Fixed a bug in URL generation method in client module that prevented certain
+  URLs from being reached.
+* Fixed HTTP method used by `coverage` method in `client.container`.
+* Fixed bug in `CompilationOutcome.from_dict` that prevented serialised
+  compilation outcomes from being parsed.
+
+
 ## 2.1.6 (2018-05-15)
 
 ### Features

README.md

@@ -1,43 +1,128 @@
 # BugZoo
 
- [![Join the chat at https://gitter.im/BugZoo/Lobby](https://badges.gitter.im/BugZoo/Lobby.svg)](https://gitter.im/BugZoo/Lobby?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
- [![Build Status](https://travis-ci.org/squaresLab/BugZoo.svg?branch=master)](https://travis-ci.org/squaresLab/BugZoo)
-
-BugZoo is an ongoing effort to provide a series of controlled environments
-for performing experiments on automated program repair of C programs. BugZoo
-provides a number of bug scenarios, adapted from existing publicly-available
-datasets, including [ManyBugs](http://repairbenchmarks.cs.umass.edu/),
-[GenProg's earlier benchmarks](http://dijkstra.cs.virginia.edu/genprog/), and
-several datasets from the
-[Software Infrastructure Repository](http://sir.unl.edu/).
-
-As well as providing the files necessary to study each bug,
-BugZoo also provides a minimal, isolated environment for execution, in the
-form of an accompanying Docker container. By providing a container for
-each bug, rather than a single, monolithic virtual machine for a group of bugs,
-BugZoo incurs fewer overheads (resulting in a much higher performance, crucial
-for rigorous experimental trials), makes dependencies easier to manage, and
-results in a smaller disk footprint.
-
-For more information on [Docker](https://www.docker.com/), the interested user is
-referred to
-[https://docs.docker.com/learn/](https://docs.docker.com/learn/).
+[![Build Status](https://travis-ci.org/squaresLab/BugZoo.svg?branch=master)](https://travis-ci.org/squaresLab/BugZoo)
+[![PyPI version](https://badge.fury.io/py/bugzoo.svg)](https://badge.fury.io/py/bugzoo)
+
+BugZoo is a decentralised platform for distributing,
+reproducing, and interacting with historical software bugs. BugZoo is designed
+to support both software engineering researchers as well as developers of tools
+for program testing, analysis and repair.
+
+Under the hood, BugZoo uses [Docker](https://www.docker.com/) containers to
+provide a controlled, sterile environment for safely interacting with
+historical software bugs. BugZoo provides several advantages over
+the use of monolithic virtual machines
+(e.g., [ManyBugs VM](http://repairbenchmarks.cs.umass.edu/)):
+
+* By using containerisation instead of virtualisation, BugZoo can ensure
+  reproducibility with minimal sacrifice to performance, allowing you to run
+  bigger experiments faster.
+* Since containers can be spawned in under a second, a safe, idempotent
+  execution environment can be realistically provided for every test suite
+  evaluation.
+* Each historical bug is provided as its own minimal Docker container image.
+  There are no assumptions that bug containers should be capable of running
+  other software, such as a testing or repair tool that is being evaluated as
+  part of an experiment. BugZoo avoids the inevitable "DLL hell" that arises
+  from attempting to mix bugs and tools by transforming tools into portable
+  binaries and mounting them in the execution environment at run-time.
+* Thanks to their minimal nature and Docker's layer-based image caching,
+  prebuilt datasets of historical bugs can be downloaded quickly and
+  easily.
+
+
+## Features
+
+* A simple but powerful command-line interface for building, downloading and
+  interacting with historical bugs.
+* A Python interface for complex, structured interactions with historical
+  bugs: execute specific tests, compute line coverage, apply patches, compile
+  with certain flags, and more.
+* A powerful underlying RESTful API that can be used to add BugZoo bindings to your
+  own favourite language.
+* A decentralised architecture that lets you share your tools and datasets of
+  historical bugs on the BugZoo platform without giving up ownership or control.
+  Adding BugZoo support to your dataset of tool is as simple as adding a
+  `.bugzoo.yml` file to your project.
 
 ## Installation
 
-To install the latest stable version of BugZoo:
+We recommend using [pipenv](http://pipenv.org/) to install BugZoo, although you
+may simply use `pip3` instead:
+
+```
+$ pipenv install bugzoo
+```
+
+BugZoo supports Python >= 3.5.
+
+## Getting Started
+
+Since BugZoo is a decentralised platform for sharing and interacting with
+historical bugs (*and not itself a dataset of bugs*), you'll need to add
+several bugs and tools to your installation. Bugs and tools are usually
+provided to BugZoo by remote, public Git repositories. Adding a *source* of
+bugs or tools to your installation can be done via the `bugzoo source add
+[name] [url]` command:
+
+```
+$ bugzoo source add manybugs https://github.com/squaresLab/ManyBugs
+```
+
+Note that sources do not necessarily have to be remote, public Git
+repositories: they can also be provided by local directories.
+
+```
+$ git clone https://github.com/squaresLab/ManyBugs path/to/manybugs
+$ bugzoo source add another-manybugs path/to/manybugs
+```
+
+The `bugzoo source list` command can be used to provide a list of registered
+sources:
+
+```
+$ bugzoo source list
+
+Source    Type    URL                                         Location                                                        Version
+--------  ------  ------------------------------------------  --------------------------------------------------------------  ---------
+robust    Local   -                                           /home/chris/bugs/robust                                         -
+genprog   Remote  https://github.com/squaresLab/genprog-code  /home/chris/.bugzoo/sources/github_com_squaresLab_genprog-code  b4de21d7
+manybugs  Remote  https://github.com/squaresLab/ManyBugs      /home/chris/.bugzoo/sources/github_com_squaresLab_ManyBugs      ad83bd9a
+```
+
+And `bugzoo bug list` and `bugzoo tool list` can be used to produce a list of
+all bugs and tools that are provided by your registered sources:
 
 ```
-$ pip3 install bugzoo
+$ bugzoo bug list
+
+Bug                                             Program     Dataset    Source    Installed?
+----------------------------------------------  ----------  ---------  --------  ------------
+manybugs:gmp:13420-13421                        gmp         manybugs   manybugs  No
+manybugs:gmp:14166-14167                        gmp         manybugs   manybugs  No
+manybugs:gzip:2009-08-16-3fe0caeada-39a362ae9d  gzip        manybugs   manybugs  No
+manybugs:gzip:2009-09-26-a1d3d4019d-f17cbd13a1  gzip        manybugs   manybugs  No
+manybugs:gzip:2009-10-09-1a085b1446-118a107f2d  gzip        manybugs   manybugs  No
+manybugs:gzip:2010-01-30-fc00329e3d-1204630c96  gzip        manybugs   manybugs  No
+manybugs:gzip:2010-02-19-3eb6091d69-884ef6d16c  gzip        manybugs   manybugs  No
+manybugs:libtiff:2005-12-14-6746b87-0d3d51d     libtiff     manybugs   manybugs  No
+manybugs:libtiff:2005-12-21-3b848a7-3edb9cd     libtiff     manybugs   manybugs  No
+manybugs:libtiff:2005-12-27-6f76e76-5dac30f     libtiff     manybugs   manybugs  No
+manybugs:libtiff:2006-02-23-b2ce5d8-207c78a     libtiff     manybugs   manybugs  No
+manybugs:libtiff:2006-02-27-6074705-e6d0c32     libtiff     manybugs   manybugs  No
+manybugs:libtiff:2006-03-03-a72cf60-0a36d7f     libtiff     manybugs   manybugs  No
+...
 ```
 
-To install the latest development version of BugZoo, first produce a shallow
-clone of this repository before executing `pip3` as shown below.
+To interact with a particular bug or tool, you'll need to install it on your
+machine. There are two ways to do this: (1) you can build the Docker image for
+the bug or tool locally (via `bugzoo bug build` and `bugzoo tool build`), or
+alternatively, you can download a prebuilt image (via `bugzoo bug download`
+or `bugzoo tool download`).
 
 ```
-$ git clone git://github.com/squaresLab/BugZoo --depth 1 bugzoo
-$ cd bugzoo
-$ pip3 install . --upgrade
+$ bugzoo bug build manybugs:libtiff:2005-12-14-6746b87-0d3d51d
+...
 ```
 
 ## Known Sources
@@ -54,3 +139,19 @@ To add one of these sources, simply execute the following from the command line:
 $ bugzoo source add manybugs https://github.com/squaresLab/ManyBugs
 $ bugzoo source add genprog https://github.com/squaresLab/genprog-code
 ```
+
+## Citation
+
+If you use BugZoo in your research work, we would highly appreciate it if you
+cited the [following paper](http://www.cs.cmu.edu/~clegoues/docs/timperley-icse18-poster.pdf):
+
+```
+@inproceedings{bugzoo,
+  author = {Timperley, Christopher and Stepney, Susan and {Le Goues}, Claire}
+  title = {{Poster: BugZoo – A Platform for Studying Software Bugs}},
+  booktitle = {{International Conference on Software Engineering}},
+  series = {{ICSE '18}},
+  year = {2018}
+  note = {{To appear}}
+}
+```

bugzoo/__init__.py

@@ -7,3 +7,4 @@
 
 import logging
 logging.getLogger(__name__).addHandler(logging.NullHandler())
+logging.getLogger(__name__).setLevel(logging.INFO)

bugzoo/cli/__init__.py

@@ -1,5 +1,6 @@
 from typing import List, Optional, Dict
 import sys
+import os
 import argparse
 import logging
 from operator import itemgetter
@@ -528,10 +529,11 @@ def build_parser():
 def main():
     try:
         log_formatter = logging.Formatter('%(levelname)s:%(name)s:%(asctime)s: %(message)s')  # noqa: pycodestyle
-        log_to_stdout = logging.StreamHandler()
-        log_to_stdout.setFormatter(log_formatter)
-        logging.getLogger('bugzoo').setLevel(logging.INFO)
-        logging.getLogger('bugzoo').addHandler(log_to_stdout)
+        log_filename = os.path.join(os.getcwd(), "bugzood.log")
+        log_to_file = logging.handlers.WatchedFileHandler(log_filename)
+        log_to_file.setFormatter(log_formatter)
+        logging.getLogger('bugzoo').setLevel(logging.DEBUG)
+        logging.getLogger('bugzoo').addHandler(log_to_file)
 
         parser = build_parser()
         args = parser.parse_args()

bugzoo/client/api.py

@@ -43,9 +43,8 @@ def __init__(self,
         self.__base_url = base_url
 
         # attempt to establish a connection
-        logging.info("Attempting to establish connection to %s within %d seconds",  # noqa: pycodestyle
-                     base_url,
-                     timeout_connection)
+        logger.info("Attempting to establish connection to %s within %d seconds",  # noqa: pycodestyle
+                    base_url, timeout_connection)
         url = self._url("status")
         time_started = timer()
         connected = False
@@ -66,13 +65,15 @@ def __init__(self,
                              base_url)
                 raise ConnectionFailure
             time.sleep(0.05)
-        logging.info("Established connection to server: %s", base_url)
+        logger.info("Established connection to server: %s", base_url)
 
     def _url(self, path: str) -> str:
         """
         Computes the URL for a resource located at a given path on the server.
         """
-        return urllib.parse.urljoin(self.__base_url, path)
+        url = "{}/{}".format(self.__base_url, path)
+        logger.debug("transformed path [%s] into url: %s", path, url)
+        return url
 
     def handle_erroneous_response(self,
                                   response: requests.Response
@@ -86,49 +87,34 @@ def handle_erroneous_response(self,
             UnexpectedResponse: if the response cannot be decoded to an
                 exception.
         """
+        logger.debug("handling erroneous response: %s", response)
         try:
             err = BugZooException.from_dict(response.json())
         except Exception:
             err = UnexpectedResponse(response)
         raise err
 
-    def get(self,
-            path: str,
-            *,
-            json: Optional[Any] = None
-            ) -> requests.Response:
+    def get(self, path: str, **kwargs) -> requests.Response:
         url = self._url(path)
-        logger.info('GET: %s', url)
-        return requests.get(url, json=json)
-
-    def post(self,
-             path: str,
-             *,
-             json: Optional[Any] = None
-             ) -> requests.Response:
+        logger.debug('GET: %s', url)
+        return requests.get(url, **kwargs)
+
+    def post(self, path: str, **kwargs) -> requests.Response:
         url = self._url(path)
-        logger.info('POST: %s', url)
-        return requests.post(url, json=json)
+        logger.debug('POST: %s', url)
+        return requests.post(url, **kwargs)
 
     def put(self, path: str, **kwargs) -> requests.Response:
         url = self._url(path)
-        logger.info('PUT: %s', url)
+        logger.debug('PUT: %s', url)
         return requests.put(url, **kwargs)
 
-    def patch(self,
-              path: str,
-              data,
-              **kwargs
-              ) -> requests.Response:
+    def patch(self, path: str, data, **kwargs ) -> requests.Response:
         url = self._url(path)
-        logger.info('PATCH: %s', url)
+        logger.debug('PATCH: %s', url)
         return requests.patch(url, data=data, **kwargs)
 
-    def delete(self,
-               path: str,
-               *,
-               json: Optional[Any] = None
-               ) -> requests.Response:
+    def delete(self, path: str, **kwargs) -> requests.Response:
         url = self._url(path)
-        logger.info('DELETE: %s', url)
-        return requests.delete(url, json=json)
+        logger.debug('DELETE: %s', url)
+        return requests.delete(url, **kwargs)

bugzoo/client/container.py

@@ -9,6 +9,7 @@
 from ..core.coverage import TestSuiteCoverage
 from ..testing import TestCase, TestOutcome
 from ..cmd import ExecResponse
+from ..exceptions import BugZooException
 
 logger = logging.getLogger(__name__)  # type: logging.Logger
 
@@ -144,17 +145,16 @@ def compile(self,
         Raises:
             KeyError: if the container no longer exists.
         """
-        path = "containers/{}/compile".format(container.uid)
-        r = self.__api.post(path)
-
+        path = "containers/{}/build".format(container.uid)
+        params = {}
+        if verbose:
+            params['verbose'] = 'yes'
+        r = self.__api.post(path, params=params)
         if r.status_code == 200:
             return CompilationOutcome.from_dict(r.json())
-
-        if r.status_code == 404:
-            raise KeyError("container or test case not found")
-
         self.__api.handle_erroneous_response(r)
 
+    build = compile
 
     def test(self,
              container: Container,
@@ -187,16 +187,21 @@ def coverage(self,
         uid = container.uid
         logger.info("Fetching coverage information for container: %s",
                     uid)
-        r = self.__api.get('containers/{}/coverage'.format(uid))
+        r = self.__api.post('containers/{}/coverage'.format(uid))
         if r.status_code == 200:
             jsn = r.json()
             coverage = TestSuiteCoverage.from_dict(jsn)  # type: ignore
             logger.info("Fetched coverage information for container: %s",
                         uid)
             return coverage
-        logger.error("Failed to fetch coverage information for container: %s",
-                     uid)
-        self.__api.handle_erroneous_response(r)
+        try:
+            self.__api.handle_erroneous_response(r)
+        except BugZooException as err:
+            logger.exception("Failed to fetch coverage information for container %s: %s", uid, err.message)  # noqa: pycodestyle
+            raise
+        except Exception as err:
+            logger.exception("Failed to fetch coverage information for container %s due to unexpected failure: %s", uid, err)  # noqa: pycodestyle
+            raise
 
     def exec(self,
              container: Container,